Category Archives: JSON

Saving Game Data in SCORM

27 Sep 2014

Every once in a while I have to wrap something in SCORM. Its not terribly often but it happens maybe about twice a year. This time around it was a Flash-based training module for a local hospital. Applying the SCORM wrapper has become rather blasé; I apply the wrapper, tell the devs what JS functions to call, tell them how to get data out of the LMS, test and then deploy.

Well, this latest training module contained unique features such as mini-games, related data, and user-selectable items. We needed to be able to save all of this data into the LMS and then retrieve the data and hand it off to the SCO so that not only would it resume from where it left off but that the previous game state would also be reflected within its UI.

The plan here was to convert the data to JSON and then store it in cmi.suspend_data. In SCORM 1.2 the suspend_data element has a maximum capacity of 4,096 characters – plenty for our needs.

Storing the data

My function for receiving the JSON payload from Flash and storing it looked similar to this:

// The data is already stringified within the Flash movie and then passed to this function
function storeSuspendData(str){
     LMSSetValue('cmi.suspend_data',str);
}

Immediately this caused an issue which turns out is due to the double-quotes that are part of JSON formatting. In order to be able to successfully insert JSON into the LMS I had to replace all the double-quotes with some other character. I chose to use an asterisk.

My function then turned into something like this:

// The data is already stringified within the Flash movie and then passed to this function
function storeSuspendData(str){
    LMSSetValue('cmi.suspend_data',str.replace(/\"/g,'*'));
}

Here’s some sample JSON:

{"icon":"car","color":"red",{"quizzes":[{"quiz":1,"score":75},{"quiz":2,"score":85},{"quiz":3,"score":95}]}}

And here’s the psuedo-JSON after the search and replace:

{*icon*:*car*,*color*:*red*,{*quizzes*:[{*quiz*:1,*score*:75},{*quiz*:2,*score*:85},{*quiz*:3,*score*:95}]}}

This allowed me to store the data within cmi.suspend_data without issue.

Retrieving the data

Somewhere in all of this an extra set of double quotes were added to the JSON – to the beginning and to the end of the string so that in reality the psuedo-JSON looked like this in the LMS:

*{*icon*:*car*,*color*:*red*,{*quizzes*:[{*quiz*:1,*score*:75},{*quiz*:2,*score*:85},{*quiz*:3,*score*:95}]}}*

My function to get the data out of the LMS at first didn’t account for the beginning and trailing asterisk:

function getSuspendData(str){
    var val = LMSGetValue('cmi.suspend_data').replace(/\*/g,'"");
    return val;
}

Which gave me some bad JSON (note the leading and trailing double-quotes):

"{"icon":"car","color":"red",{"quizzes":[{"quiz":1,"score":75},{"quiz":2,"score":85},{"quiz":3,"score":95}]}}"

So I had to clean that up:

function getSuspendData(str){
    var val = LMSGetValue('cmi.suspend_data').replace(/\*/g,'"');
    val = val.substr(1,val.length);
    val = val.substr(0,val.length - 1);
    return val;
}

When I ran a JSON.parse on my now valid JSON and logged it to the console I could see a bonafide JS object.

Passing the JSON to Flash

The next step was to provide the JSON to flash and we decided to use FlashVars but that came with a problem. Flash vars is simply HTML and HTML attributes use double-quotes. In the sample below you can see the problem right away when the HTML for the Flash object was written to the page:

<param name="FlashVars" value="gameData="{"icon":"car","color":"red",{"quizes":[{"quiz":1,"score":75},{"quiz":2,"score":85},{"quiz":3,"score":95}]}}"" />

Obviously JSON’s double quotes are breaking the HTML. The solution was to not do the search/replace of the previous asterisks – I changed things so that Flash would receive the pusedo-JSON via the FlashVar which meant that the string clean-up would have to happen within Flash.

My updated “get” function looked similar to this:

function getSuspendData(str){
    var val = LMSGetValue('cmi.suspend_data');
    val = val.substr(1,val.length);
    val = val.substr(0,val.length - 1);
    return val;
}

And the FlashVar became valid HTML:

<param name="FlashVars" value="gameData=*{*icon*:*car*,*color*:*red*,{*quizes*:[{*quiz*:1,*score*:75},{*quiz*:2,*score*:85},{*quiz*:3,*score*:95}]}}*" />

And then finally in Flash I simply did the search and replace on the asterisk back to double-quotes and then did a JSON.parse to transform the string into a useful object.

JSON is a String

26 Nov 2013

I had a developer once tell me that the Sencha Touch project we were working on was “just JSON”. I had not considered that the code we were writing had a resemblance to JSON until that moment. My head being consumed with the project required me to supply a shrug of the shoulders and go back to my work… but later I realized that no, thats not the case at all and IMO it was not only a gross over simplification of what we were doing at the time but also showed a lack of understanding of what JSON is.

JSON is a string. It is also an acronym – JavaScript Object Notation. JSON looks like this:

'{
   "name" : "groupname",
   "type" : "some type",
   "description" : "some description"
}'

As noted above we should be looking at a string and we are by virtue of the surrounding single quotes. When we do an AJAX request that returns JSON we receive something like the above example. It is common practice to then use some library to parse the JSON response into a JavaScript Object. If we don’t then the JSON string is of little use. Therefore many libraries/frameworks, such as jQuery or Sencha Touch, have JSON parsers to convert JSON into a bonafide JavaScript Object.

Modern browsers have parsing built-in natively via two two useful methods: JSON.parse() and JSON.stringify() where the former converts (deserializes) JSON into an object and the later converts (serializes) an object into a JSON-formatted string. The name of that last last method – **stringify()** – should IMO end the discussion of what JSON is (if the official JSON spec didn’t already do that) – JSON is a string. A string with a specific format but a string none-the-less.

Why then does the term JSON get used so liberally? Lets do a JSON.parse() on our JSON example. This is what we get:

{
   name : "groupname",
   type : "some type",
   description : "some description"
}

What we are looking at here is our converted JSON – now a JavaScript object – represented via a JavaScript Object Literal. It *looks* like “JavaScript Object Notation” (JSON) but obviously it is not. Yet the label is applied anyway. The logic appears to be that if it looks like JSON but doesn’t necessary “smell” like JSON it must somehow be JSON despite the fact that “notation” is an inert facsimile of an object whereas a “literal” is literally the object in question.

jQuery.parseJSON vs JSON.parse vs json_parse – Memory Profiles During AJAX Requests

10 Apr 2013

When doing an AJAX JSONP request with jQuery you rely on the jQuery.parseJSON method to parse the JSONP response into an object. While jQuery.parseJSON works most of the time I’ve seen it behave as if a memory leak is occurring. This isn’t new to jQuery.ajax() as it has somewhat of a history with memory leaks. The version I am using currently is jQuery 1.9.0, and with a particular project I am seeing a “stair-step” memory leak pattern where the usual ebb and flow of memory being allocated/deallocated appears as a predictable pattern. However, at a certain point there is a spike that itself never goes down which in effect sets the normal ebb/flow memory usage pattern at a higher level than previous. This happens 4 times within my web app, each spike setting a higher and higher average level of memory usage.

The Application

To understand what this is all about I should give a brief description of this particular application. What I have is an app that will be wrapped in PhoneGap. When this app launches on a mobile device it will download an XML document. That document contains data referencing all of the app’s downloadable assets. The app will then check to see if the data exists locally or not (if it did it would have been stored on the app via WebSQL) and if not (or there has been an update as determined by each asset’s timestamp) it will determine what files need to be downloaded and commence the download operation.

The “download” consists of AJAX calls to individual JSONP-type documents which have a structure similar to {date:value,name:value,data:value} where “data” is an image serialized to base64.

There are hundreds of these asset files. The individual files allow for the creation of a download/progress indicator where the file that is currently being downloaded is of course a percentage of the total.

The rest of this discussion deals with the downloading of the application’s assets as that is where the memory issue occurs.

jQuery.parseJSON

Anyone who uses jQuery.ajax() who works with JSON/JSONP will (or at least *should*) be using jQuery.parseJSON. The jQuery framework will take the JSON/JSONP server response and transparently transform it into a JavaScript Object to do with as the developer sees fit.

Initially I implemented the AJAX request exactly as I’ve done many times before not seeing the need to deviate from what has been a predictable pattern of interacting with JSONP server responses. Here is a code snippet of such a setup:

...
$.ajax({
   url:       server_path_here,
   type:      'GET',
   dataType:  'JSONP',
   timeout:    30000
...

Simple stuff, we’re requesting a JSONP response from the server and we have a timeout of 30 seconds.

Observe the following memory usage timeline obtained from Chrome via a script using the above AJAX configuration. The script “loops” in a synchronous fashion through an array of URL’s in an effort to cache the data within the browser. The workflow is: make request > download > parse > store via WebSQL > on txn success/failure move to next URL and start the process over:

jquery_memory_leak_complete_download

You can see quite plainly the pattern here and it is evident that we have a memory leak. Upon investigating the timeline I can see the exact AJAX calls where the leaks occur. The data contained within the JSONP payloads is the aforementioned base64 string and their lengths exceed 2 million characters. I now know the culprit but not why the leak happens but suspect that there is something wrong with jQuery’s handling of the data, in particular, jQuery.parseJSON.

JSON.parse

Some Googling on the topic reveals that there is some previous history on memory leaks while using jQuery to do AJAX calls. One note in particular mentioned the eval method being the root cause. jQuery’s JSON parser is based on work by Douglas Crockford who is well known within the JavaScript community. All web app developmers should be aware of him and his JSON parser script as it is commonly used/referenced by many frameworks. Anyway, I know from having visited D Crockford’s site many times that he offers more than one JSON parser and that one of them explicitly avoids using the eval method in its parsing routines.

Before I proceed to json_parse (the “eval-less” parser) I want to try JSON.parse. It follows that his JSON.parse script will also exhibit the memory leak when presented with JSON exceeding 2 million characters (and thus by extension any JavaScript library’s JSON parser based on his work, of which there are many). Here is the config for that test:

...
$.ajax({
   url:       server_path_here,
   type:      'GET',
   dataType:  'JSONP',
   timeout:    30000,
   converters: {'text json':JSON.parse} 
...

And the memory profile While using JSON.parse:

json.parse_memory_leak_complete_download

We can see the same thing happening – time to use Douglas Crockford’s alternative parser – json_parse.

json_parse

Douglas Crockford describes json_parse as “an alternative JSON parse function that uses recursive descent instead of eval.” Perfect – we want to get away from eval, here’s its implementation within jQuery:

...
$.ajax({
   url:       server_path_here,
   type:      'GET',
   dataType:  'JSONP',
   timeout:    30000,
   converters: {'text json':json_parse} 
...

And the resulting memory usage:

json_parse_memory_leak_non_optimized_complete_download

Thats what we’re looking for – you can see the spikes as the multi-million character JSON payloads get parsed and you can see GC happening afterwards. Memory gets reclaimed just the way we want it to.

From here we can delve further into optimizing the code responsible for these graphics – which is what I’ve done. A week or so of optimization and in some cases code-rewrite results in the following final download memory footprint for my web app:

json_parse_memory_leak_complete_download

Through streamlining a number of things – including removing all my attempts at debugging the issue – I’ve cut the download time and the overall memory foot print has been further improved – possible in large part to json_parse.

Conclusion

jQuery.parseJSON is useful for the vast majority of applications. However, if you are dealing with large JSON payloads – in excess of 1.8 to 2+ million characters in length you should use an alternative JSON parser such as json_parse. Do so and you will ensure that memory is reclaimed as needed and thus avoid app slugishness and crashes.

Here’s the link to get both JSON.parse and json_parse: https://github.com/douglascrockford/JSON-js.

Convert XML to JSON

27 Nov 2012

Having a library of scripts in your back pocket comes in handy – this one is a gem IMO, it is an XML to JSON converter written in JavaScript. I spent the better part of my day trying the wide variety of scripts that a simple Google search yielded. Some didn’t work, others did albeit with quirks (i.e., output that was “technically” JSON while having a less than desired structure).

After much searching and testing I’ve found one that is “perfect”; inserting null values where needed, arrays appearing where they should, extra objects weren’t inserted, etc, etc…. in addition the author shared his test scripts to show the robustness of his work. Download it from here:

http://michael.hinnerup.net

The library claims to also convert JSON to XML – I’ve not tried it but if the quality of the XML to JSON portion is any indication then I expect it should work just as well.

Update 1/2/2013

Michael appears to have updated his blog as all of the content has vanished – and so as a service to others (and myself) I’m providing a link to the js. Whenever Michael gets his blog backup I’ll revert to linking to him, in the meantime you can download the library here:

xml_to_json.js

Also, as his documentation has also disappeared I’ve added a quick example below showing how to request an XML resource via JQuery and turn it into JSON once its been received by the client:

...
function getXML(){
  $.ajax({
    url:      hostURL,
    type:     'GET',
    dataType: 'xml',
    error:function(xhr,status,err){
      // handle errors here
    },
    success:function(responseData,status,xhr){
      if (responseData.childNodes.length > 0){
        var obj = xmlJsonClass.xml2json(responseData); // xml to json conversion happens here
        console.log(obj);
      } else {
        console.log('Response did not appear to contain any children.');
      }
    }
  });
}
...

Tip

I had XML that contained CDATA sections – what the XML-to-JSON script did was create a needless object for the CDATA – a structure that looked like this:

...
{test:
   {test1:'something',
    test2:'something else',
    test3:{#cdata:'another something'}
    }
}
...

Note the value for “test3” – instead of putting the string “another something” as the value the library put in another object. To avoid that, open the xml_2_json.js file and go to line 186 and note the for loop:

...
for (n = xml.firstChild; n; n = n.nextSibling) {
   o["#cdata"] = this.escape(n.nodeValue);
 }
...

That block of code is where the extra object gets inserted – simply change it to this:

...
for (n = xml.firstChild; n; n = n.nextSibling) {
   //o["#cdata"] = this.escape(n.nodeValue); // << original line
   o = this.escape(n.nodeValue); // << new line
 }
...

And now the sample output above appears in a manner that makes more sense:

...
{test:
   {test1:'something',
    test2:'something else',
    test3:'another something'
    }
}
...