Creating a chroma key effect with HTML5 canvas.

If you have a video/photograph that has a plain background, such us you’d find if you used a green/blue screen or maybe on a plain sky background, you can create a chroma key effect. This is the same technique used by weather presenters and on hollywood movies to replace the background with something else.

We’ll go through some of the trials an tribulations of using this to replace the background of an image and even on a video.

Technologies Used:

Demo

Link

Libraries used in this demonstration:

Browser support:

Should be supported on all HTML5 browsers, however, on iOS devices, you cannot transfer video frames to canvas elements (argh!).

Step 1: Set up your image and canvas

First thing’s first, we’ve got to include our base image and create a canvas to serve as our platform for our chroma key effect. Inserting the image is done in the usual way:

<img id="source" src="img/chromakey/manwithbriefcase.jpg" alt="A man with a briefcase against a greenscreen background" />

Then we’ll create our canvas using javsacript:

var $canvasbg = $("<div id='target-container' />");
var canvas = $("<canvas class='greenscreen id='target' />").get(0);
$canvasbg.append(canvas);
$('body').append($canvasbg);

We’ve also created a containing div here so that we can easily change the background colour without having to update the whole canvas.

Step 2: Put our image on the canvas

Now we use the drawImage() function to put our image on the canvas. We’ll do this on the image’s onLoad event so that all the image data is loaded before we try to do anything with it. Note, we’ll do some resizing of the canvas and container as well:

var context = canvas.getContext('2d');
canvas.width = img.clientWidth;
canvas.height = img.clientHeight;
$canvasbg.width(img.clientWidth);
$canvasbg.height(img.clientHeight);
context.drawImage(img, 0, 0);

With all luck, you should have something that looks like this:

Greenscreen
Greenscreen

Step 3: Loop through the image data and turn green pixels transparent

To make the green pixels transparent, we’re going to loop over every pixel in the image and read it’s RedGreen, Blue and Alpha values. If the pixel is green, then we’ll set it’s alpha channel value to zero.

var imageData = context.getImageData(0, 0, canvas.width, canvas.height);
var data = imageData.data;
var start = {
    red: data[0],
    green: data[1],
    blue: data[2]
};

// iterate over all pixels
var tolerance = 150;
for(var i = 0, n = data.length; i < n; i += 4) {
    var diff = Math.abs(data[i] - data[0]) + Math.abs(data[i+1] - data[1]) + Math.abs(data[i+2] - data[2]);
    if(diff < tolerance) {
        data[i + 3] = 0;
    }
}
context.putImageData(imageData, 0, 0);
$canvasbg.css('background','#ABC123');

As can be seen, we’ve got the green colour from the very top left pixel in the image. This gives us the target colour we’re trying to remove. We then look at the difference in pixel values compared to that top right pixel and, if the difference is small enough, we remove it.

After we’ve looped over every pixel, we then draw the image back on the canvas. Finally, we change the background colour of our canvas container to a colour of your choice. If all goes well, you should end up with something like this:

First try at chroma key

This is good for our first try, however there are one or two key improvements.

Step 4: Create some semi transparent pixels

The previous code just switched pixels on or off if the difference was at a certain threshold. Instead, I suggest we update the algorithm so that the level of transparency is related to the level of difference between the two pixels. The creates a slightly softer edge to our cutout. After a bit of playing around, I found that this gave satisfactory results:

var tolerance = 150;
for(var i = 0, n = data.length; i < n; i += 4) {
    var diff = Math.abs(data[i] - data[0]) + Math.abs(data[i+1] - data[1]) + Math.abs(data[i+2] - data[2]);
    data[i + 3] = (diff*diff)/tolerance;
}

And hopefully now, you should be getting something like this:

Updated algorithm
Updated algorithm

Slightly better than our original try, and probably about as good as we’re going to get with this size image.

Step 5: Extend, extend, extend

There are several things we could do to extend our little algorithm.  For one, this method isn’t limited to images. We can draw on the canvas with HTML5 video elements as well. In fact, I’ve used this method to create a pre-rendered 3d model viewer:

Conclusion

This is a great way to have a green screen type of effect using HTML5 canvases, however, looping over pixels is an expensive operation. If you’re doing this with video or anything that is displaying live, you’ll find at a certain size this method is just too slow for most uses.

Saying that, you never know when you might need to create a greenscreen effect.

12 Replies to “Creating a chroma key effect with HTML5 canvas.”

    1. Indeed, however I could see it’s use in dynamic replacement of parts of images, for example showing a logo in a certain place or replacing the contents of a computer screen in an image or similar. Static images it should be fine for.

  1. Hello, I´m working on a Picture Cabin, where you can choose a backgroud, like the beach or snow, then you take yourselft a picture to finally print it out. You think is posible to do it using this? how can I use de web cam and your algorithm to do it?.
    Thaks so much

Leave a Reply to adminCancel reply