Image Bluescreen

< Code In The Browser

Building on the earlier image manipulation and logic sections, here we see how to implement a movie special effect.


We have played with logic to detect just specific color areas in an image .. what could we do with those areas? The earlier stop-sign examples set the area to some fixed color.

The "bluescreen" (also known as "chroma key") technique in movie special effects, (a) selects areas in an image by color as we do above, and (b) changes those pixels to pixels from another image. This is how many movie special effects are done (a video is just a series of images shown in succession over time).

Suppose we have two images, "image" with the stop sign, and "back" with a pattern of green leaves.

The following bluescreen example code detects the red part of the sign, and for those pixels, copies over the pixels from the back image. We'll look at these lines in detail to see how it works. The bluescreen code below has 3 key differences from the earlier examples:

  1. back = new SimpleImage("leaves.jpg"); -- open a second image and store it in the variable "back" (thus far we've only opened one image at a time)
  2. pixel2 = back.getPixel(x, y); -- say we are looking at pixel x,y in the main image. Get the x,y pixel from the other image and store it in the variable "pixel2". We can get the x and y of the current pixel in the loop with the functions pixel.getX() and pixel.getY().
  3. pixel.setRed(pixel2.getRed()); -- copy the red value from pixel2 to pixel. Also do this for the other two channels. In effect, for the x,y location, this copies copies the pixel from back to image.

Copy pixel code:

    x = pixel.getX();
    y = pixel.getY();
    pixel2 = back.getPixel(x, y);

Experiments: (a) Try copying the leaves to the sky instead of to the red parts of the sign (similar to the earlier Night Mode problem). (b) Make the code a little shorter by eliminating the need for "x" and "y" variables. Put the calls to pixel.getX() and pixel.getY() right in the back.getPixel(...) call.

Abby Bluescreen

In this example we have the abby.jpg image -- make it appear that the green parts of her little chair are sprouting leaves. (Solution at end of doc).

1. Monkey Moon Exercise

Now you do one just like Hollywood. Here we have monkey.jpg and moon.jpg. Select the blue background pixels behind the monkey, and for each such pixel, copy over the corresponding pixel from the moon.jpg; the result is the monkey hanging out on the moon. The blue background behind the monkey is a little dark, however the monkey (brown and yellow) has very little blue in it, so you can figure out a * factor that separates the monkey and its blue background.

Experiments: We also have paris.jpg and yosemite.jpg -- try using those as the background instead of moon.jpg. Note that your if-logic does not look at the background at all ... the detection logic is all on the monkey image, so there's no limit about the colors or whatever in the background. That is, there's lots of blue in paris.jpg but that does not mess things up.

The paris.jpg image is much larger than the monkey image. We end up just using the upper-left quarter of paris.jpg. The image code supports a setSameSize() function which can be used to scale an image up or down in size to match the size of another image. Call it like this to scale the paris.jpg back image to be the same size as the monkey image: back.setSameSize(image); -- the monkey will appear with the full size paris or yosemite or whatever background image.

Previously we relied on the fact that the background images are at least as large as the foreground images. If they backgrounds were smaller without resizing -- say the monkey image was 400 pixels wide, but the moon image was only 300 pixels wide -- we would get an out-of-bounds error when trying to copy non-existent pixels from the moon image. If you have that problem, use setSameSize() as show above.

2. Totoro Exercise (optional)

Here we have the totoro.jpg image of the character from the movie My Neighbor Totoro, (BTW a great kid movie from the creator of Spirited Away). Write code to show Totoro on the moon or paris or yosemite. The edges won't come out quite a perfect as the monkey example, since Totoro's bottom has a bit of a green/yellow cast, reflected off the cloth when I took the picture.

Custom Bluescreens (optional)

Now it's time to try the bluescreen computation with your own images. To do this, you need to "base64" encode your foreground and/or background image. Base64 encoding takes image data and encodes it as a long stream of random looking characters. (This Base64 step is a consequence of doing all the computation in the browser; we are running into certain security constraints of the browser, and the Base64/aux1 technique shown below is the way to work with the browser.)




Some solution code.

Abby Leaves Bluescreen:

  if (pixel.getGreen() > avg * 1.15) {
    pixel2 = back.getPixel(pixel.getX(), pixel.getY());