So during the lightning talks on Friday I was feverishly writing NumPy code with the hope of getting it running for my demo at 9am on Saturday. At about 1 am Matt and I were finally convinced that it worked and went back to the hotel room, where I built another pair of IR glasses until 2 am.

This did not leave time for altering my presentation, so I left it out. I did mention it briefly after demoing the 3D game, but I would guess that many listeners missed it.

In any case, I basically reimplemented the method I was using to filter for red pixels in NumPy, but now the system scans every pixel of every frame. Also, rather than returning a single point where the laser is “seen” it returns a list of points if there are multiple lasers.

The basic process is that I first look at just the R value of the RGB tuple. You can filter out pixels that don’t have a high enough R value very quickly.

Once I have a list of candidate pixels I look at them in more detail, calculating their redness.

Then I define redness as:

redness = R*2 – G – B

Thus a white pixel would have a redness of zero. A pure red pixel would have a redness of 510 (255*2-0-0). Again, NumPy makes this process fast, but it isn’t nearly as fast as the first filter due to the fact that it includes a multiply and two adds. In fact, I tried to run it without the first filter (which doesn’t affect results, only speed) and it slowed things down a lot.

Finally I sort the list of pixels by redness and then begin putting those pixels in a results list. But if a pixel is within about 15 pixels of a pixel already in the results list it is assumed to be part of the same laser and excluded. Currently I am also capping the return list to a length of 5, which also helps with performance in some instances.

Here’s the code as it currently stands, with what I hope are sufficiently copious comments.

#! /usr/bin/env python -t
'''
isightlaser

scans for laser and manages list of laser hits
This is a poorly named class as it works with a vartiety
of webcam inputs, not just the iSight

'''

import numpy

# The cameras we've tested so far all have 640x480 resolution
CAMERA_MAX_X = 640
CAMERA_MAX_Y = 480
NUM_PIXELS = CAMERA_MAX_X * CAMERA_MAX_Y

# for RGBA data this is the matrix we multiply by to get redness
MUL_ARRAY = numpy.array([2,-1,-1,0])

# some libraries return bitmaps as BGRA ordered data, so
# this is the matrix we use to get redness
BGRA_MUL_ARRAY = numpy.array([-1,-1,2,0])

class IsightLaser(object):

def __init__(self, top=0, bottom=480, left=640, right=0, game_xy=(1024,768)):
self.queue = []
self.top = top
self.bottom = bottom
self.left = left
self.right = right
self.game_xy = game_xy
self.calibrate = True

# called when game surface is sized or resized
def set_game_xy(self, xy_pair):
self.game_xy= (xy_pair)

def pop(self):
if self.queue:
return self.queue.pop()
else:
return None

def has_elements(self):
return self.queue

def push(self, xy_pair, flip_y=False):
# any pixels seen during calibration are used to define
# the limits of what the camera can see
if self.calibrate:
print "calibrating"
x,y = xy_pair
if y > self.top:
self.top = y
if y < self.bottom:
self.bottom = y
if x > self.right:
self.right = x
if x < self.left:
self.left = x

else:
trans_pair = self.translate(xy_pair, flip_y)

self.queue.append(trans_pair)
print self.top, self.bottom, self.left, self.right, self.game_xy

def translate(self, xy_pair, flip_y=False):
# translate a point from camera coordinates to game world coordinates
x = self.game_xy[0] * (xy_pair[0] - self.left)/(self.right-self.left)

# some cameras have their origin in the upper left, others in the lower left
if flip_y:
y = self.game_xy[1] * (xy_pair[1] - self.top)/(self.bottom - self.top)
else:
y = self.game_xy[1]- self.game_xy[1] * (xy_pair[1] - self.top)/(self.bottom - self.top)
#print xy_pair , x,y
return (x,y)

def start_calibrate(self):
self.calibrate =True
self.top = 0
self.bottom = 480
self.left = 640
self.right = 0

def stop_calibrate(self):
print "STOP CAL"
self.calibrate = False

def process_buffer(self, image_buffer, rgba=4, flip_y=False, bgra=False):
global MUL_ARRAY
# convert from buffer type to numpy array
flat_array = numpy.frombuffer(image_buffer,numpy.uint8)

# convert from a (640*480*4) x 1 matrix to a (640*480) x 4 matrix
image_array =numpy.reshape(flat_array,(NUM_PIXELS,rgba))

# filter out one colum of the matrix in order to just get red values
if bgra:
reds = image_array[:,2]
else:
reds = image_array[:,0]
#print  "image arrary:", image_array
#print "reds", reds
#image_array =numpy.multiply(image_array,MUL_ARRAY)

# Filter out any red value that is less than 180 (should be adaptive later)
mask = numpy.greater(reds,180)
#print mask

# get indices of pixels with a red value over 180
original_indices = numpy.array(mask.nonzero())[0]
#print "original_indices", original_indices

# pixel_list is an n x 4 matrix of candidate pixels
pixel_list = numpy.array(image_array[original_indices])
#print "pixel list: ", pixel_list
if bgra:
MUL_ARRAY = BGRA_MUL_ARRAY

# next two lines multiply and then sum in order to get
# redness = 2*R - G - B + 0*A
pixel_list = numpy.multiply(pixel_list,MUL_ARRAY)
#print "after mul", pixel_list
pixel_list = pixel_list.sum(axis=1)
#print "after sum", pixel_list

# filter for redness over 300
pixel_indices = numpy.greater(pixel_list,300).nonzero()
#print "pixel ind", pixel_indices
red_pixels = numpy.array(pixel_list[pixel_indices])
#print "red pixels", red_pixels

# Work back to get the index of pixels that pass both
# filters
original_indices = original_indices[pixel_indices]
#print "original_indices", original_indices

# create an n x 2 matrix with redness values and indicies
combined_array = numpy.column_stack((red_pixels,original_indices))
#print "ca", combined_array

# sort by redness
combined_array = numpy.sort(combined_array,axis=0,kind='quicksort')
#print "sorted ca", combined_array
clean_list = []

# take reddest pixel first, put it in the list
# then take next reddest pixel and put it
# in the list if it isn't too close (sqrt(200) pixels)
# to a pixel already in the list
for pixel in combined_array[::-1]:  # this reverses the order from the sort
#print "pixel:", pixel
if clean_list.__len__() > 5: #limits total number of dots tracked
break

x = pixel[1]%640
y = pixel[1]/640

add_elem = True
for clean_elem in clean_list:
if ((x-clean_elem[0])**2 + (y-clean_elem[1])**2) < 200:
add_elem = False
break
if add_elem:
clean_list.append((x,y))

#print clean_list
for elem in clean_list:
self.push((elem[0],elem[1]), flip_y)

laser_singleton = IsightLaser()

The code is here and includes a READ ME file, the text of which I’ve pasted below:

READ ME

IMPORTANT:
Educate yourself about laser safety before using laser pointers with this game or for any project.

http://www.osha.gov/SLTC/laserhazards/

http://en.wikipedia.org/wiki/Laser_safety

More info and help available at:

http://insightvr.com

http://blog.insightvr.com

Laser Controlled Missile Command-like Game

This game demonstrates using PyGame and PySight to create a laser controlled video game.

The missileCommand.py file contains the bulk of the game and provides a cross-platform, mouse controlled PyGame implementation of something resembling Missile Command. It requires that you have Python and PyGame installed.

Run it at the command line with:

python missileCommand.py

The laser control code is currently Mac OS X only. It requires an iSight camera. This can be a built-in iSight in a laptop or iMac. It could also be an external iSight or any external fire-wire camera. I have only tested with my built-in iSight and with a Sony DV camera.

I believe that running with laser control requires the following to be obtained and properly installed:

Red laser pointer(s)
a projector or other external screen that is not connected to your iSight
PySight from http://livingcode.blogspot.com/2005/10/pysight-preview.html
CocoaSequenceGrabber from http://www.skyfell.org/cocoasequencegrabber.html
NumPy
PyObjC
PyGame

Setup of the projector and camera isn’t too hard, but I’ll share some hints. First, it needs to be dark. The darker the better. Second, the camera needs to have the entire screen in its field of view. Note that it doesn’t need to fill its field of view. You can check this quickly and easily by running PhotoBooth. Finally you can do rear-projection or front projection, but I believe that the camera needs to be on the same side of the screen as the projector. However the players do not. Thus on Halloween I have my computer and projector safely in the garage, a sheet which serves as a screen in the door of the garage, and kids playing with lasers in the driveway.

Run the laser controlled version of the game with the following command line statement:

python main.py

I presented Marshie Attacks at PyCon 2008 today. This provided some motivation for getting the project posted online and whatnot. So I’ve put up this blog and also a very primitive project page at InsightVR.com to try to share this stuff.

Feel free to post comments and ask questions and I’ll do my best to be responsive. Hopefully I’ll be able to post more details, videos, and explanations as time goes on.