Category Archives: jQuery

jQuery.parseJSON vs JSON.parse vs json_parse – Memory Profiles During AJAX Requests

10 Apr 2013

When doing an AJAX JSONP request with jQuery you rely on the jQuery.parseJSON method to parse the JSONP response into an object. While jQuery.parseJSON works most of the time I’ve seen it behave as if a memory leak is occurring. This isn’t new to jQuery.ajax() as it has somewhat of a history with memory leaks. The version I am using currently is jQuery 1.9.0, and with a particular project I am seeing a “stair-step” memory leak pattern where the usual ebb and flow of memory being allocated/deallocated appears as a predictable pattern. However, at a certain point there is a spike that itself never goes down which in effect sets the normal ebb/flow memory usage pattern at a higher level than previous. This happens 4 times within my web app, each spike setting a higher and higher average level of memory usage.

The Application

To understand what this is all about I should give a brief description of this particular application. What I have is an app that will be wrapped in PhoneGap. When this app launches on a mobile device it will download an XML document. That document contains data referencing all of the app’s downloadable assets. The app will then check to see if the data exists locally or not (if it did it would have been stored on the app via WebSQL) and if not (or there has been an update as determined by each asset’s timestamp) it will determine what files need to be downloaded and commence the download operation.

The “download” consists of AJAX calls to individual JSONP-type documents which have a structure similar to {date:value,name:value,data:value} where “data” is an image serialized to base64.

There are hundreds of these asset files. The individual files allow for the creation of a download/progress indicator where the file that is currently being downloaded is of course a percentage of the total.

The rest of this discussion deals with the downloading of the application’s assets as that is where the memory issue occurs.

jQuery.parseJSON

Anyone who uses jQuery.ajax() who works with JSON/JSONP will (or at least *should*) be using jQuery.parseJSON. The jQuery framework will take the JSON/JSONP server response and transparently transform it into a JavaScript Object to do with as the developer sees fit.

Initially I implemented the AJAX request exactly as I’ve done many times before not seeing the need to deviate from what has been a predictable pattern of interacting with JSONP server responses. Here is a code snippet of such a setup:

...
$.ajax({
   url:       service_path_here,
   type:      'GET',
   dataType:  'JSONP',
   timeout:    30000
...

Simple stuff, we’re requesting a JSONP response from the server and we have a timeout of 30 seconds.

Observe the following memory usage timeline obtained from Chrome via a script using the above AJAX configuration. The script “loops” in a synchronous fashion through an array of URL’s in an effort to cache the data within the browser. The workflow is: make request > download > parse > store via WebSQL > on txn success/failure move to next URL and start the process over:

jquery_memory_leak_complete_download

You can see quite plainly the pattern here and it is evident that we have a memory leak. Upon investigating the timeline I can see the exact AJAX calls where the leaks occur. The data contained within the JSONP payloads is the aforementioned base64 string and their lengths exceed 2 million characters. I now know the culprit but not why the leak happens but suspect that there is something wrong with jQuery’s handling of the data, in particular, jQuery.parseJSON.

JSON.parse

Some Googling on the topic reveals that there is some previous history on memory leaks while using jQuery to do AJAX calls. One note in particular mentioned the eval method being the root cause. jQuery’s JSON parser is based on work by Douglas Crockford who is well known within the JavaScript community. All web app developmers should be aware of him and his JSON parser script as it is commonly used/referenced by many frameworks. Anyway, I know from having visited D Crockford’s site many times that he offers more than one JSON parser and that one of them explicitly avoids using the eval method in its parsing routines.

Before I proceed to json_parse (the “eval-less” parser) I want to try JSON.parse. It follows that his JSON.parse script will also exhibit the memory leak when presented with JSON exceeding 2 million characters (and thus by extension any JavaScript library’s JSON parser based on his work, of which there are many). Here is the config for that test:

...
$.ajax({
   url:       service_path_here,
   type:      'GET',
   dataType:  'JSONP',
   timeout:    30000,
   converters: {'text json':JSON.parse} 
...

And the memory profile While using JSON.parse:

json.parse_memory_leak_complete_download

We can see the same thing happening – time to use Douglas Crockford’s alternative parser – json_parse.

json_parse

Douglas Crockford describes json_parse as “an alternative JSON parse function that uses recursive descent instead of eval.” Perfect – we want to get away from eval, here’s its implementation within jQuery:

...
$.ajax({
   url:       service_path_here,
   type:      'GET',
   dataType:  'JSONP',
   timeout:    30000,
   converters: {'text json':json_parse} 
...

And the resulting memory usage:

json_parse_memory_leak_non_optimized_complete_download

Thats what we’re looking for – you can see the spikes as the multi-million character JSON payloads get parsed and you can see GC happening afterwards. Memory gets reclaimed just the way we want it to.

From here we can delve further into optimizing the code responsible for these graphics – which is what I’ve done. A week or so of optimization and in some cases code-rewrite results in the following final download memory footprint for my web app:

json_parse_memory_leak_complete_download

Through streamlining a number of things – including removing all my attempts at debugging the issue – I’ve cut the download time and the overall memory foot print has been further improved – possible in large part to json_parse.

Conclusion

jQuery.parseJSON is useful for the vast majority of applications. However, if you are dealing with large JSON payloads – in excess of 1.8 to 2+ million characters in length you should use an alternative JSON parser such as json_parse. Do so and you will ensure that memory is reclaimed as needed and thus avoid app slugishness and crashes.

Here’s the link to get both JSON.parse and json_parse: https://github.com/douglascrockford/JSON-js.

Cache Busting Your Web Service Requests

29 Jun 2012

While working with web services you always need to keep in mind that web browsers cache files. if you web service request doesn’t change ever so slightly you will run into the issue of the web browser not pulling the new response but instead pulling the previously cached response. This is easily mitigated by adding a variable that itself is always unique – notice that i didn’t say “random” – random may work, sure, but “random” means that you run the risk of encountering that number again at some point. Yes, feel free to come up with a random 20 digit number and it may be for all intents and purposes appear to be “unique” but the risk of that number re-appearing is still there, however obscure it may be. As most web service calls are GET requests its as simple as adding a new name/value pair with a unique value to every request. The service ignores it and the browser takes note that the request is different than the previous request and so accepts the new response from the server.

Add the following somewhere within your code to get the unique value:

...
   var d = new Date();
   var unique = d.getTime();
...

That gives you a number similar to 1340979406482. That value is the number of milliseconds since January 1, 1970 so naturally it increments every millisecond. Ideally the getTime() method would accept a date as an argument and would then provide the milliseconds between Jan 1, 1970 and the specified date but since we don’t provide that value the current date is assumed. So, that number represents the number of milliseconds from that date in 1970 and the millisecond in time when the code was executed.

I have seen people blog this topic but include dividing the millisecond value by 1000 and rounding the result. Not a great idea. Dividing by 1000 and rounding the result shortens the length of the number by 3 digits. If you shorten it too much it won’t be useful. Already, by dividing by 1000 I’m not getting a number that updates every millisecond but instead a number that updates every second. I suppose it depends on your idea of “unique” and your specific application, but I’d rather have the guarantee of uniqueness than have something that is fuzzy like doing Math on the already guaranteed unique number… makes no sense.

As to applying this, here’s a jQuery example:

...
var d = new Date();
var unique = d.getTime();
var dataStr = 'name_value_pairs_here';
$.ajax({
  url:      	'/some_service_here/',
  data:      	dataStr + '&nocache=' + unique,
  contentType:	'application/x-www-form-urlencoded',
  dataType: 	'json',
  error:function(obj,status,xhr){
    // handle error here
  },
  success:function(obj,status,xhr){
    // handle response here
  }
});
...

And better yet – jQuery has such a thing built into it, simply set the cache attribute as in this example:

...
var dataStr = 'name_value_pairs_here';
$.ajax({
  url:      	'/some_service_here/',
  data:      	dataStr,
  cache:        false, // cache buster
  contentType:	'application/x-www-form-urlencoded',
  dataType: 	'json',
  error:function(obj,status,xhr){
    // handle error here
  },
  success:function(obj,status,xhr){
    // handle response here
  }
});
...

When using the “cache” attribute within the jQuery Ajax config object jQuery will append a new name/value pair to the GET request in the form of:

  1. _=[timestamp]

Where “[timestamp]” is a unique value.

The usefullness of using d.getTime() is still there outside of jQuery. For example any call to a file that you don’t want to be cached, such as web pages, JS files, CSS, etc.

Cross-Domain Ajax and XML with jQuery

07 May 2012

Did a fun little thing today – built a jQuery/PHP-based Cost of Living (COL) calculator for a city’s website (sorry, have to be vague on the client’s name) utilizing the data from www.coli.org.

Coli.org provides a service that allows interested parties to compare the cost of living across a range of criteria between two different locations within the Untied States.

Typically as in this case the destination location is the city that is providing the calculator. The intent is to allow potential new residents to compare their current cost of living to the destination city.

To start with you will need to query two of the services on page load – one to get the list of origin locations and another to get the list of destination locations (typically one or two locations – and if you want only one destination you could easily hard code the value of the city via a hidden form field or similar). Once you receive this data you will need to populate two select lists to allow the user to make their selections. With that data you will then query another web service to get the results. Further manipulation of the returned data on the client side with some math will get you percentages and adjusted income.

Approach

There are a couple of ways to approach this – server-side or client-side. As I’m a front-end guy my approach then was to do Ajax knowing that jQuery makes it trivial to do AJAX and parse XML. However, since this was going to be a cross-domain call a 100% client-side solution wouldn’t be possible. Instead, I would have to use something that wouldn’t be subject to a browser’s cross-domain policy – a proxy to act as the go-between for client and the web service. The server the site is hosted on supports PHP and so that dictated the tech to use. A little research revealed PHP’s cURL and a few lines of code later I had myself the proxy that I needed to accomplish the cross-domain service calls.

Proxy

One thing I noticed in the live examples that Coli.org provided is that the GUID was exposed which seemed unnecessary to me. So, instead of following their implementation examples verbatim I placed the GUID into the PHP proxy thereby keeping it secure. Of course, Coli.org might be mitigating GUID abuse by restricting calls to registered domains – I’ve no idea, but in any event, it seemed to be a better approach to not expose it if possible.

If you’re looking into using the Coli.org data by mirroring my approach you’ll be interested in the proxy, which I’ve provided below:

...

define ('HOSTNAME', 'http://www.coli.org/_clicktocomparews/service.asmx/');
$path = $_GET['ws_path'];
$url = HOSTNAME.$path;
$session = curl_init($url);
if ($_GET['ws_path']) {
     $getvars = '';
     while ($element = current($_GET)) {
          $getvars .= key($_GET).'='.$element.'&';
          next($_GET);
     }
     $getvars .= '&licenseeGUID={_YOUR_GUID_GOES_HERE_}';
     curl_setopt ($session, CURLOPT_POST, true);
     curl_setopt ($session, CURLOPT_POSTFIELDS, $getvars);
}

curl_setopt($session, CURLOPT_HEADER, false);
curl_setopt($session, CURLOPT_RETURNTRANSFER, true);
$xml = curl_exec($session);
header("Content-Type: text/xml");
echo $xml;
curl_close($session)

...

All you need to do is to replace the string “_YOUR_GUID_GOES_HERE_” with the GUID that Coli.org gives you and as long as your server supports cURL your proxy is ready to be used. Note that the curly braces surrounding the numeric GUID is necessary.

AJAX & jQuery

Next is the AJAX. jQuery really makes it easy to do AJAX, here’s a stripped down version of my AJAX call to the Proxy:

...
function doAjax(wsPath,args){
   var dataStr = args ? 'ws_path=' + wsPath + '&' + args : 'ws_path=' + wsPath;
   $.ajax({
      url:      '/col_proxy.php',
      data:      dataStr + '&action=compute',
      dataType: 'xml',
      error:     function(xhr,status,err){
         //console.log(status + '\n' + err);
      },
      success:function(responseData,status,xhr){
         // console.log('success')
      });
}
...

So lets break this down, doAjax() expects two arguments to be passed to it. “wsPath” is the web service path, “args” are the arguments to be sent to the webservice. You would use the success callback to process the returned XML.

The web service paths for Coli.org’s web services are the following (as can be publicly seen here):

  • GetComparisonDetail
  • GetComparisonSummary
  • GetFromCommonPlaceTotal
  • GetFromLocations
  • GetPeriodInfo
  • GetSampleLocations
  • GetToCommonPlaceTotal
  • GetToLocations

Since I need the list of origin and destination locations first before any other web service can be used I call doAjax() via $(document).ready with the following placed at the end of my script:

...
$(document).ready(function(){
... 
... 
  // bottom of script
  (function init(){
    doAjax('GetFromLocations',null);
    doAjax('GetToLocations',null);
  })();
}); 

The “init” function is self-invoking and as you might have guessed, “GetFromLocations” is the web service for getting the list of origin locations, and “GetToLocations” provides the list of possible destination locations. These two services don’t require any extra attributes so the second argument to doAjax() is simply passed null.

The next task would be to call another webservice once the origin and destination cities have been selected by passing the data that repesents the city selections and the desired web service path. I didn’t use all of the webservices that Coli.org has availiable as they weren’t needed for my application. For my needs only two more calls were needed: “GetComparisonDetail” and “GetComparisonSummary”. In both of these cases the required name-value pairs to send along are:

  • fromCity
  • toCity

So then, knowing this and we would would pass the web service name as the first argument for doAjax() and construct the name/value pair string to be the value of the second argument. Something that in your own application may look similar to this:

...
// put together the "from" and to" name value pairs
// where "fromVal" is the origin city code and
// "toVal" is the destination city code
var args = 'fromCity=' + fromVal + '&toCity=' + toVal;

// specify the web service to use
var wServiceName = 'GetComparisonSummary';

// query the webservice
doAjax(wServiceName,args);
...

Processing XML

The result from a successful web service call will be XML. Coli.org unfortunately doesn’t provide XML samples so you’ll have to discover this on your own. What I did was to construct the entire web service call **sans proxy** directly in my browser’s address bar. The entire call, from www.colie.org… all the way to the GUID and any other necessary arguments – and then hit enter on my keyboard. If succesfull the XML response will appear in-browser.

Here’s an example of discovering the XML structure via a direct link in a browser:

http://www.coli.org/_clicktocomparews/service.asmx/GetFromLocations?licenseeGUID={_YOUR_GUID_HERE}

Here is a portion of the returned XML:

...

  
  T-Bone SteakFairbanksAK110.439.2655586592178789Q12011
T-Bone SteakWarsawIN19.79.2655586592178789Q12011
Ground BeefFairbanksAK13.83.0240502793296091Q12011
...

Once you know the XML structure the next few steps couldn’t be easier – in this case, each location is described as an XML node/tag called “Table”. Within this node are other nodes that themselves contain the specific values that you are looking for, such as:

  • Category_Name
  • Place_Name
  • State
  • Cost

In jQuery to find any node/tag within an XML doc simply use the “find()” method and you will get an array of all matching nodes. looking back at the doAjax() sample code we can see that the success callback assigns the returned XML to “responseData”. So then we have all the piecess to this rather small puzzle. To find all of the “Table” nodes within “responseData” our jQuery code would be:

...
var tableArray = $(responseData).find('Table');
...

This gives us all the “Table” nodes – but… we of course have to loop through each one looking for the specific values that we need and while a for loop would work here to iterate through each “Table” we should instead utilize chaining and jQuery’s built-in iterator, the “each()” method, and its callback:

...
$(responseData).find('Table').each(function(){
  // additional processing instructions here;
});
...

Pretty cool how chaining allows us to do multiple things simultaneously, right? This is a rather small chaining example too, but anyway, the next step is to find the desired values within each individual “Table” node. This is quite simple as well, see the next example:

...
$(responseData).find('Table').each(function(){
  var cname = $(this).find('Category_Name').text();
  var pname = $(this).find('Place_Name').text();
});
...

The $(this) statement in this context simply means “the current Table node” – then we look for the specific child node inside the current Table by again using the find() method. And on the end, we use the “text()” method to pull the text that is within the desired nodes. Here we are pulling the text for the “Category_Name” and “Place_Name” nodes which again are child nodes of “Table”.

THis is great, but this code doesn’t do us much good at this point as all we end up with are the Category and Place values of the very last Table in two variables. Instead we would actualy use this to populate the page with HTML for each individual Table. We can quickly print out the results to the page by appending html to a pre-existing div. As seen in the following example our existing div is called “tableResults”:

...
$(responseData).find('Table').each(function(){
  var currentRow =  '

Category: ' + $(this).find('Category_Name').text(); currentRow += ', Location: ' +$(this).find('Place_Name').text() + '

'; $('#tableResults').append(currentRow) }); ...

This will print out the Category and Location of each Table node within the mentioned pre-existing div called “tableResults”. Note that I’ve wrapped each set of Table data in a P tag. This gives us some automatic padding between lines which makes things a little more visually appealing. This sample is pretty basic, you’ll of course want to do more in your own application.

And… (drum roll)… thats about it. This was just a brief overview of the technical aspects of integrating into Coli.org’s web services via client-side technologies, not a complete functional example though with the code samples above – and complete working PHP-based proxy – you should be able to integrate with their web services quite easily or at the very least get a jump start on a similar project of your own.

Logo

All content © 2012-2017.