CSC 249/449 Computer Vision: Video Surveillance Assignment

The goal of this project is to track people entering the field of view of a camera, and for each track, acquire a frontal picture of the face of the person generating the track, if possible, and the best picture of the face in any case. The system should run in real time, which means in practice, at least 4-5 frames a second in order to make tracking the people feasible. In order to get image data from the frame buffer fast enough, you will want to run the system at reduced resolution, (e.g. one half or one quarter). Once inside the machine, you may find it makes sense to run different operations at different resolutions both for efficiency, and algorithmic reasons.

The first step is to detect moving pixels in the image. In this world, we will assume that anything moving is a person. The easiest way to do this detection is by a technique known as background subtraction. Basically, you have a reference image that represents the background, and any pixel in the current image that differs significantly from the background is marked. (Assumption: it must have moved in there from somewhere). This is really change detection rather than motion detection, but in indoor environments (and even outdoors with a little massaging) the two phenomena are highly correlated.

Issues include: What constitudes "significant" difference. You can use a fixed threshold, but it is also possible to use a spatially varying threshold (or more complex model) that will permit the system to ignore fluttering leaves or flickering video displays locally. Such models are usually learned on-line by letting the system observe the background for a bit. Another issue is updating the reference image. Outdoors, and even indoors, there are occasional global changes to the reference image, due to change in lighting, or motion of the camera. Most systems re-acquire the background; either continuously using exponential time averaging, or periodically. If you do this, it makes sense not to update the reference image in any location where a moving object is currently being detected. You can also check for global changes (lots of separated locations changing simultaneously) indicating either camera movement or rapid lighting change, and shut down change detection during these periods, and then quickly re-acquire a reference image when global motion stops.

The second step is to assign detected pixels to groups representing individual objects (represent objects by bounding rectangles). Simple connected components is likely to fail here, since most objects will show up as several pieces. A step that groups e.g., blobs that are closer to each other than, say, half the size of the biggest blob in the group (or some other fraction) could be used, as could other clustering heuristics. This is one step that it makes sense to run at reduced resolution, since you don't need accurate segmentation - just a bounding box, and connected components operations are relatively expensive.

In the third step the objects (bounding boxes) have to be tracked from frame to frame. This is easy if there is only one object; however if there are several, correspondences must be identified. A little bit of spatial proximity and prediction works well here. It is important to remember that objects can disappear from the scene (even in the middle of the picture (they go behind something), and also merge and split. You are not required to handle all these event correctly, but they should be handled robustly in that they don't cause the system to break (for example, the merging of two object might be handled by killing both tracks and starting a new one, killing one track and keeping one, or by trying to keep track of the fact that two objects are in a blob; but it should not cause the transfer of one of the tracks to a distant bit of noise.

The final step is to get a face shot for each track. Best way of doing this is probably to periodically grab a window from the top of each object (presumeably containing the head); evaluate it using some "faceness" measure (based on color or patterns), and replace the old picture if the current one has a higher face score. When the track terminates, it has the best face found associated with it.

Your system should work in real time, displaying boxes around the tracked objects (either on the video screen or on the computer monitor). It should also record trajectories for each tracked object. Printed output should give the time and location of the beginning and end of each track, and show the acquired picture. As usual, a significant portion of the grade will be based on a careful writeup of the project, describing the issues, algorithms, performance, and breaking points.

Back to vision course main page