Category Archives: html5

Diffing Two Canvases

03 Sep 2016

Below is the script I wrote that will take two canvases and find the differences between them – a canvas diff. What it returns is the bounding area – upper left coordinates and lower-right coordinates – with which you can do as you like.

The challenge here is iterating through the pixel data. The data itself is an array of each pixel’s RGBA values in sequence. For example, if we are looking at four pixels then the pixel array representing them would have a length of 16 (RGBA = 4 array elements x the number of pixels = 16). This imaginary array could look like:

  • 0,0,122,5,100,12,123,6,16,100,43,123,55,55,100,50

With a little formatting to help make it make more sense we can easily see the groupings:

  • 0,0,122,5,    100,12,123,6,    16,100,43,123,    55,55,100,50

Looking at the first group of 4 numbers we can see that the pixel they represent has these RGBA values:

  • R: 0
  • G: 0
  • B: 122
  • A: 5

Imagine a 1024×768 image represented by a single data array of RGBA values. Such an image would have a data array length of 3,145,728 (1024 x 768 x 4). In order to manipulate the pixel data you’d have to discover a way of looping through the array keeping in mind that every group of 4 array elements is itself a single pixel. You would also need to realize that any given group of array elements represents a specific pixel somewhere within your image/canvas.

Image Comparison Example

In the example shown here I’m comparing the pixel data between two images, keeping track of the differences, and returning a bounding area that encompasses them.

This example is in an iframe – view source to see the script that loads the images into image objects, then loads those images into two canvas elements, extracts the image data from the canvases, compares them, and then finally draws the red bounding boxes to illustrate where the images diverge from each other.

Diff Function – Comparing Two Canvases

The function I wrote below compares all the pixels and notes the coordinates of the ones that don’t match. I then use the sort array method to sort the resulting “diff” array by either the X or Y of each diff’d coordinate so that I can find the extremes of each one.

While I’m looking for the differences I am also keeping track of the X and Y coordinate representing each RGBA quadruplet. Note the use of the modulus operator as that is what makes the coordinate-tracking work.

To use this function all you have to do is create two image data objects using the canvas getImageData() method then pass them to the canvasDiff function where the first data object is the original image and the second is the one that has changed. Refer to the iframed example above – view source to see how the diff function seen below was used within the example to produce the bounding differential boxes.

When using images as the source data they need to be of identical size and ideally in PNG format. PNG is optimal because JPG is a lossy compression algorithm that will make it hard to do a legit diff – provided you are using images at all – you could easily just have programmatically-generated canvas art – point is that canvasDiff needs two image data objects where the images are the same physical dimensions.

function canvasDiff(imageData1,imageData2){
    // www.rickluna.com - please leave the attribution!
    var w = imageData1.width;
    var h = imageData1.height;
    var diffs = [];
    var start = {x:null,y:null};
    var end   = {x:null,y:null};
    var pA1_r,pA1_g,pA1_b,pA1_a,
	pA2_r,pA2_g,pA2_b,pA2_a;
    var y = 0;
    var x = 0;
var len = imageData1.data.length;
    for (var i=0;i b.x){
	    return 1;
	} else {
	    return 0;
	}
    });
    start.x = diffs[0].x || 0;
    end.x = diffs[diffs.length-1].x || w;
    diffs.sort(function(a,b){
	if (a.y < b.y){
	    return -1;
	} else if (a.y > b.y){
	    return 1;
	} else {
	    return 0;
	}
    });
    start.y = diffs[0].y || 0;
    end.y = diffs[diffs.length-1].y || h;

    // DONE
    // "start" and "end" have the bounding coordinates
    console.log(start,end);
    return [start,end];
}

CanvasPainter

27 Jul 2016

One of the ways a developer shows the passion he or she has for their work is how they spend their free time. I do my best to spend my time learning more about what I like to do. Being a fellow who has a degree in Graphic Design that means that I have spent a lot of time over the years slaving over self-assigned projects in an effort to learn the things that I never learned in college.

A few years ago I wrote a canvas signature widget for a PhoneGap’d Sencha Touch-based mobile app. That tiny canvas signature pad was the genesis for the thing that I today call CanvasPainter. While it grew slowly over the ensuing years via bursts of productive energy it mostly languished in the dusty corners of hard-drives and USB memory sticks. That has changed and a lot of time is now being applied to CanvasPainter.

The collage at the top of this page illustrate portions of the CanvasPainter UI – portions of a fully functional web app. A group of beta testers are currently running it through its paces.

You will be able to experience CanvasPainter online. If you are inclined, you will eventually be able to buy it from the App Store. The hybrid app version will do things that the web app version does not. What those features will be I’ll share at a later date as I get closer to launch.

Anyway, this page serves as my way of sharing my excitement at the approaching V1.0 milestone (huzzah!! applause!!!).

[edit 7/27/2016]

Below is a sample image that I made with CanvasPainter… as a result of having painted this I realized I needed a color picker and that the swatches needed to be modified a little, which lead to a new thing in the settings panel…

made with CanvasPainter

Made with CanvasPainter

[edit 10/26/2016]

CanvasPainter is now available for purchase at the App Store. Took less than 48hrs for it to be accepted – apparently they didn’t find any issue with it. Visit the app’s website at www.canvaspainter.io to learn more about what it does.

I will also do a breakdown here at my website running through how the app was made as time allows.

Installing Crosswalk in an Older PhoneGap Project

08 Sep 2015

Starting with Android 5.0 the webview has been separated from the OS and is now itself an app that will receive updates like any other app. The implication here is that from Android 5.0 and up users won’t have to receive an OS update to get better webviews for PhoneGap apps to use, they can now download and install WebView updates separate from the OS.

All fine and good – in the meantime older Android versions still have the webviews that shipped with the OS / device. OS updates don’t happen very often, typically only occurring when someone gets a new phone or until the device manufacturer decides to issue an update. In even rarer instances a device might be updated if it is rooted and a custom ROM is applied.

Until recently this meant that PhoneGap developers would still have to deal with the various webkits that their apps may encounter and whatever features might be lacking (The most infamous example of a lacking feature that comes to mind was the completely missing toDataURL() canvas method a few years ago.)

Given all the different Android devices in the wild it is sometimes quite a task for developers to be able to create hybrid apps that behave predictably in all the different webkit flavors. The Crosswalk plugin addresses this issue by embedding a web runtime in your Android apps (Android 4 and up) meaning that you don’t need to worry about the version of webkit on older Android OS’s as you now have an modern embedded Chromium under the hood giving you a consistent environment for your apps to execute within.

Today I’m trying to update an older app of mine so that it will use Crosswalk and I hit an issue that prevented me from installing it. The error:

“Plugin doesn’t support this project’s cordova-android version. Cordova-android: 3.6.3, failed requirement: >=4”

corsswalk_1

The first thing to be aware of is that the CrossWalk plugin requires Android 4. While searching online revealed some hacks to force Crosswalk to install that wasn’t something that I wanted to do.

My first instinct was to edit the manifest.xml since I recall from having previously used Eclipse to compile my apps that that would be the course of action to take. Unfortunately as you may know the Eclipse ADT no longer works with Eclipse and the latest SDK’s so thats not something I want to try and get working since the Cordoca CLI makes compiling a lot easier.

Anyway, editing the manifest.xml didn’t seem to have an effect as rebuilding did nothing and trying to re-add the platform resulted in the “Platform android already added” error message. Searching online revealed a few hacks that could be done and seems to have worked for some people but I didn’t want to pursue any hacky methods so I stopped bothering with trying to use my current project files.

In the end I decided to rebuild my PhoneGap project and target a specific version of Android so as to meet Crosswalk’s requirements. The takeaway is that yes, you’ll have to rebuild and wont be able to install CrossWalk for apps targeting pre Android 4 devices.

The easiest way to rebuild the project to target more recent versions of Android is to simply delete the Android folder out of the project and then rebuild it. While a wholesale delete works a cleaner method (presumably) is to issue the CLI command to remove the platform. Either way you don’t have to worry about reinstalling all of your plugins as Cordova will detect their prior installation and install them all for you when the new target platform is created.

So then, I deleted the Android folder using the CLI via this command…

  • cordova platform remove android

…and then reinstalled Android making sure to target a specific version using this syntax:

  • cordova platform add android@x.x.x

Where the @x.x.x represents the desired version of the OS.

An easy way to see what version numbers will work is to try to create an Android project with a version that doesn’t exist. Try Android 4.2 and Cordova will of course fail the attempted platform add but also show you the valid version numbers / install targets that it expects.

corsswalk_2

That’s a nice reference – I decided to go for Android 4.1.1 JellyBean.

  • cordova platform add android@4.1.1

In the image below you can see the result of the install followed by a list of all the installed plugins – note that it happened on its own since if you were to look in the project folder all the plugins are there – all we did was delete the platform so Cordova still knows what plugins to install.

corsswalk_3

A quick build showed my app working – all done!

Creating Image Maps From Canvas-Derived Coordinates

10 Jun 2015

Here’s a cool thing – I came across a situation where I was stacking identically-sized transparent PNG’s on top of each other but needed to be able to select their visible areas. The layered nature of z-ordering the images prevented us from getting beyond the top-most layer (actually, not image tags but z-ordered divs with background-images, but for all intents and purposes its the same issue).

The first thought was to use HTML5 Canvas and then track the coordinate of the users click/touch event to figure out what they were clicking on. A nice start but our browser requirements included old IE which prevented us from using Canvas so we were stuck with the PNG stack. In addition to dealing with old IE the requirements of the project meant also that we couldn’t simply merge all the PNG’s and then simply create an image map because:

  • Our client would be uploading the UI-related PNG’s into the system – both on and off states
  • We couldn’t rely on the client to be smart enough to draw image maps within the admin console each time an image was uploaded, especially since some of the art bumped or “went under” some of the other art within the PNG stack.

I thought we could still use Canvas if we applied it to the process of uploading the images – knowing that we essentially had small bitmaps “floating” within a larger transparent PNG I realized that we might be able to get some useful data from a Canvas, maybe enough to determine what should be clickable and what shouldn’t.

A quick Google search revealed this post at stackoverflow. It describes a “Marching Squares Edge Detection” algorithm that when applied to my needs would give me an array of coordinates that could be easily converted into an image map.

Detecting Multiple Edges

Looking at the edge detection algorithm showed that it detects the edge for the first shape that it encounters, ignoring anything else in the Canvas even though other shapes may exist. The fix here was to remove the shape that it finds and then run the algorithm again, repeating the find/remove process until nothing else is found. This is done while saving the boundary of each shape so that when the entire process was done we could use that data to create the image maps.

As for removing the first shape that was encountered I wrote the following – it draws a shape on the canvas exactly where the found shape is using that shape’s boundary and then overwrites it with a fill. Since I set the Canvas globalCompositeOperation to destination-out the end result is that the shape is removed from the canvas thus allowing me to find the next shape’s boundary as the previous shape no longer exists.

    function _removeBitmap(){
        var i, len;
	// draw outline path
	_ctx.globalCompositeOperation = 'destination-out';
	_ctx.beginPath();
	_ctx.moveTo(_points[0][0],_points[0][1]);
	for (i=1,len=_points.length;i<len;i++){
    	    var point = _points[i];
	    _ctx.lineTo(point[0],point[1]);
	}
	_ctx.closePath();
	_ctx.fill();
	_ctx.globalCompositeOperation = 'source-over';
    }

Detecting if the Canvas is Empty

Next, before I call the edge detection function again I first need to know if the Canvas is empty – so here’as another function that looks to see if the alpha of any pixel is set below a certain threshold. Why a threshold? Well, it seems that even though I can’t see the bitmaps that I removed with the destination-out composite operation that there may still be pixels here and there that do exist though are effectively invisible. A threshold settles that particular issue, you may need to play with it on your own if you use this code.

    ns.isEmpty = function(){
        var data = _ctx.getImageData(0,0,_canvas.width,_canvas.height).data;
        var emptyThreshold = 20; // maximum allowed alpha before a pixel is considered "empty"
        var i, l;
        var retVal = true;
        // maxAlpha: what is the highest alpha? most times its not zero.
        // log this to the console to see what the max alpha is, 
        // then set "emptyThreshold" accordingly.
        var maxAlpha = 0;  
	for (i=0,l=data.length; i < l; i += 4){
            // for debugging purposes
            maxAlpha = data[i + 3] > maxAlpha ? data[i + 3] : maxAlpha; 
		if(data[i + 3] > emptyThreshold){
                retVal = false;
	    }
	}
        return retVal;
    }

Yes, I know about loading up a blank canvas, getting its base64 via toDataURL and then trying to compare against it to see if a Canvas is empty – but note again how the composite operation left some pixels behind which means that comparing against a truly blank canvas wouldn’t work.

Working Example

Here’s a working example – inspect the iFramed page and note the absence of any image maps, then click “Start”. What you will see is:

  • Each image is loaded into the Canvas
  • The edge detection script finds the first “floating” shape, I remove it and then run the edge detection again, repeating until i determine that the image is now empty of any “solid” shapes
  • The next image is loaded and the process repeats
  • Once all edges have been found the image map is added to the DOM

NOTE: click start and let the sample run through all of the edge detection for everything (slower on mobile as all of the images load synchronously). It will be done when all the pieces display. From there you can click the other buttons.

You will note that the Canvas is still in this proof-of-concept after all of the edge detection is completed and the image maps are added – the final implementation of this stacks all images via absolute positioning without any Canvas elements. The top-most image of the stack has the image map applied to it. In this way we are able to automate the creation of the image maps within the browser when each image is uploaded into the system via the purpose-built CMS and not need to worry about the client using some sort of drawing tool to create the image maps themselves.

Finished Code

Here’s the result – separated from the edge detection stuff which I broke out into its own file that you can download from here while viewing the source of the example to learn how the parts were assembled.

;(function(ns,$){

    var _canvas, _ctx, _cw, _points, _imgData, _data;
    var _allPaths = [];
    var _img = new Image();
	_img.crossOrigin = 'anonymous';
	_img.onload = _drawImgToCanvas;

    function _drawImgToCanvas(){
	_ctx.drawImage(_img,_canvas.width/2-_img.width/2,_canvas.height/2-_img.height/2);
        _findArea();
    }

    function _findArea(){
        _imgData = _ctx.getImageData(0,0,_canvas.width,_canvas.height);
        _data = _imgData.data;
        _points = marchingSquares.contour();// call the marching ants algorithm
	_allPaths[_allPaths.length] = _points;// store the area in the _allPaths collection
	_removeBitmap();// remove the shape so we can move on to the next one
    }

    function _removeBitmap(){
        var i, len;
	// draw outline path
	_ctx.globalCompositeOperation = 'destination-out';
	_ctx.beginPath();
	_ctx.moveTo(_points[0][0],_points[0][1]);
	for (i=1,len=_points.length;i<len;i++){
	    var point = _points[i];
	    _ctx.lineTo(point[0],point[1]);
	}
	_ctx.closePath();
	_ctx.fill();
	_ctx.globalCompositeOperation = 'source-over';
        
        if (ns.isEmpty()){
            _createMap();
        } else {
            _findArea();
        }
    }
	
    ns.isEmpty = function(){
	var data = _ctx.getImageData(0,0,_canvas.width,_canvas.height).data;
        var emptyThreshold = 20; // maximum allowed alpha before a pixel is considered "empty"
        var i, l;
        var retVal = true;
        var maxAlpha = 0; // what is the highest alpha? most times its not zero. log this to console to see what the max alpha is, set "emptyThreshold" accordingly.
	for (i=0,l=data.length; i < l; i += 4){
            maxAlpha = data[i + 3] > maxAlpha ? data[i + 3] : maxAlpha; // for debugging purposes
	    if(data[i + 3] > emptyThreshold){
                retVal = false;
	    }
	}
        return retVal;
    }
	
    function _createMap(){
	var mapTPL = '%areas%';
	var areasTpl = '';
	var areas = '';
	var map = '';
	var coordsList = '';
	for (var h=0,len=_allPaths.length;h<len;h++){
 	    coordsList = '';
	    for (var i=0,len2=_allPaths[h].length;i<len2;i++){
		coordsList += _allPaths[h][i].join(',');
		coordsList += i != _allPaths[h].length -1 ? ',' : '';
	    }
	    areas += areasTpl.replace('%coords%',coordsList);
	}
	map = mapTPL.replace('%areas%',areas);
	$('#mapWrapper').html(map);
    }
    
    ns.returnData = function(){
	return _data;
    }
	
    ns.returnCW = function(){
	return _cw;
    }
	
    ns.init = function(canvasID,imgSrc){
	_canvas = document.getElementById(canvasID);
	_ctx = _canvas.getContext('2d');
	_cw = _canvas.width;
	_img.src = imgSrc;
    }
	
})(this.mapFromCanvas = this.mapFromCanvas || {}, jQuery);

Creating UI Using HTML5 Data Attributes

28 Mar 2015

As you may know part of the HTML5 spec includes the use of “data” attributes within HTML elements. For example:

<div data-animal="frog"></div>

You can see the naming convention – the first part must be “data-” and the part after can be anything. This feature provides a mechanism for storing information within your elements for whatever your use-case happens to be.

jQuery provides and easy way of accessing the HTML5 “data” attribute via the data() method as shown here:


<p data-color="red">What color am I hiding?</p>

console.log($(p).data('color')); // outputs "red" to the console

One way to use this is as part of a UI methodology that looks for specific “data” tags in response to user interaction. Thus your application will know that when the user clicks on an element that the element in question is a button (for example) and that the button should trigger specific functionality.

My personal convention for “buttons” and the like is an “action” data tag – data-action=”doSomething”. When I don bother using a framework my applications are always look for this type of attribute within elements. This quickly becomes important if you delegate events to an element higher in the DOM, for example, the body tag.

The sample code below shows this kind of setup.

<html><head><title></title>
<script>
   $(document).ready({
      $('body').on('click','div',changeSomething);

      function changeSomething(e){
         // do something
      }
   });
*lt;/script>
</head>
<body>
   <div id="wrapper">
      <div>
         <div data-action="changeColor">Random Colored Text</div>
      </div>
   </div>
</body>
</html>

All well and good – if the user clicks on the text “Randomly Colored Text” then the changeColor function is fired.

Look at this arrangement:

<html><head><title></title>
<script>
   $(document).ready({
      $('body').on('click','div',changeSomething);

      function changeSomething(e){
         // do something
      }
   });
</script>
</head>
<body>
   <div id="wrapper">
      <div>
         <div data-action="changeColor">Randomly Colored Text</div>
         <div>Hello</div>
         <div data-action="changeSize">Randomly Sized Text</div>
      </div>
   </div>
</body>
</html>

Clicking on either of the DIVs will trigger the “changeColor” function – and in fact, clicking on ***any*** div on the page will call that function – and that includes the wrapping DIVs. We need a way to discern which DIV element is actually something that we care about.

<html><head><title></title>
<script>
   $(document).ready({
      $('body').on('click','div',changeSomething);

      function changeSomething(e){
         if (!$(e.target).data('action')){
            return;
         }
         // do something 
      }
   });
</script>
</head>
<body>
   <div id="wrapper">
      <div>
         <div data-action="changeColor">Randomly Colored Text</div>
         <div>Hello</div>
         <div data-action="changeSize">Randomly Sized Text</div>
      </div>
   </div>
</body>
</html>

On line 7 we’re now checking to see if the data attribute “action” exists in the div that triggered the event. If it doesn’t exist we stop the flow of execution via the return statement. That’s a good start – lets take it one step further as we still need to know how to “changeColor” (line 18) or “changeSize” (line 20).

<html><head><title></title>
<script>
   $(document).ready({
      $('body').on('click','div',changeSomething);

      function changeSomething(e){
         if (!$(e.target).data('action')){
            return;
         }
         if ($(e.target).data('action') === 'changeColor'){
            console.log('change the fonts color here');
         }
         if ($(e.target).data('action') === 'changeSize'){
            console.log('change the fonts color here');
         }
      }
   });
</script>
</head>
<body>
   <div id="wrapper">
      <div>
         <div data-action="changeColor">Randomly Colored Text</div>
         <div>Hello</div>
         <div data-action="changeSize">Randomly Sized Text</div>
      </div>
   </div>
</body>
</html>

Now we have it – not only are we looking to see if any DIV element that has been clicked is something that we should pay attention to but we are also looking for specific data attributes from which we need to do something (lines 10 and 13). In this case we’re attempting to change the color of the text or change the size of the text.

If you elect to you could always just look for the specific data attributes and not do a check for the “action” data attribute – whatever you like to do – my personal approach is a larger topic making use of both “checks” – see jQuery, PubSubs and Event Delegation.

Something that will mess things up for you is the common practice of nesting multiple elements within a wrapping “button” DIV.

<html><head><title></title>
<script>
   $(document).ready({
      $('body').on('click','div',changeSomething);

      function changeSomething(e){
         if (!$(e.target).data('action')){
            return;
         }
         if ($(e.target).data('action') === 'changeColor'){
            console.log('change the fonts color here');
         }
         if ($(e.target).data('action') === 'changeSize'){
            console.log('change the fonts color here');
         }
      }
   });
</script>
</head>
<body>
   <div id="wrapper">
      <div>
         <div data-action="changeColor">
            <div class="homeIcon"></div>
            <div class="buttonText">
               Randomly Colored Text
            </div>
         </div>
         <div>Hello</div>
         <div data-action="changeSize">Randomly Sized Text</div>
      </div>
   </div>
</body>
</html>

The data-action attribute is still on the button wrapper as seen on line 23. The difference is that we now have two DIV elements within that wrapper – they will be the target of any click events and thus our code will skip over that button and not do anything. You may be tempted to add data-action attributes to everything inside of your button wrapper but that gets messy. Instead lets add some more code to our initial checking of whether or not the data-action attribute exists. Lets add something that iterates through the parents of the div looking to see if one of them contains the data attribute that we are looking for.

if (!$(e.target).data('action')){
    // if e.target does not have a data action, search through a max of 2 parents to find it
    var obj = e.target;
    var max = 0;
    while (!$(obj).data('action') && max != 2){
        obj = $(obj).parent();
        ++max;
    }
    if (!$(obj).data('action')){
        return;
    } else {
        e.target = obj;   
    }
}

Here’s whats happening here – the code looks first for a data-action attribute. If it is not found it then proceeds to look at the parent elements to see if they have them. If not found in one parent it then looks at that parent’s parent, and so on. We have a maximum ceiling here of 2 levels of parent searching. Plenty for this example and for most applications.

Once the parent with a data-action attribute is found the object that represents that parent is used to replace the click event’s “target” – as an FYI all event objects store the element that was clicked, holding them in the event object’s “target” property. By replacing the target – e.target – with the parent object that ***does*** contain the required data attribute we allow our code to continue what it was doing previously.

The full example:

<html><head><title></title>
<script>
   $(document).ready({
      $('body').on('click','div',changeSomething);

      function changeSomething(e){
         if (!$(e.target).data('action')){
             // if e.target does not have a data action, search through a max of 2 parents to find it
             var obj = e.target;
             var max = 0;
             while (!$(obj).data('action') && max != 2){
                 obj = $(obj).parent();
                 ++max;
             }
             if (!$(obj).data('action')){
                 return;
             } else {
                 e.target = obj;   
             }
         }
         if ($(e.target).data('action') === 'changeColor'){
            console.log('change the fonts color here');
         }
         if ($(e.target).data('action') === 'changeSize'){
            console.log('change the fonts color here');
         }
      }
   });
</script>
</head>
<body>
   <div id="wrapper">
      <div>
         <div data-action="changeColor">
            <div class="homeIcon"></div>
            <div class="buttonText">
               Randomly Colored Text
            </div>
         </div>
         <div>Hello</div>
         <div data-action="changeSize">Randomly Sized Text</div>
      </div>
   </div>
</body>
</html>