Data-Flow Diagram

How the System Works:

As shown in the data-flow diagram (above), the steps in the system are, from top to bottom:

1. The system begins with a motion sequence of n+1 frames of a person walking.
   • View an MPEG of sample sequence.

2. We compute the optical flow (Bulthoff, Little, and Poggio - source code) of the motion sequence to get n frames of (u,v) data, where u is the x-direction flow and v is the y-direction flow.

   View MPEGs of flow for sample sequence:

   • u,
   • v,
   • |(u,v)|.

   In the u and v images, red indicates negative flow while green indicates positive. To save computation time, we manually tracked each subject and computed flow only in a box surrounding the person. You can see artifacts from this box in the MPEG images of the flow sequences.

3. Spurious motion in the image background can confound recognition. The aspect ratio signals are particularly sensitive to outliers. To eliminate as many outliers as possible, we compute connected components of regions that are moving and discard the small components. The resulting moving "blobs" have less spurious motion.
   • View MPEG of blobs for sample sequence.

4. For each frame of the flow, we compute a set of scalars (see source code) that characterize the shape of the flow in that frame. Examples include x and y coordinates of the moving region (centx and centy), and the aspect ratio of the moving region (aspct).

   View the cropped motion sequence with the the position of (centx,centy) shown as a "+", the |(u,v)|-weighted centroid (wcentx,wcenty) shown as a box, a solid ellipse corresponding to aspct, and a dashed ellipse corresponding to the aspect ratio of the |(u,v)|-weighted moving region, waspct.

   • scalar sequence
   Note how the weighted ellipse (particulary senstive to fast-moving parts of the body) follows the motion of the feet as they accelerate in the gait.

5. We rearrange the scalars to form one time series for each scalar.

   View plots of some of the signals for the example sequence:

   • raw time series for centx, centy, aspct
   • centx, centy, aspct with linear background removed

6. The time series are sinusoidal and have a common frequency. We use least-squares linear prediction spectrum analysis (Barrodale and Erikson's method - source code) to find the fundamental frequency and phase for each series. A Matlab example that demonstrates the advantages of this method is available.

   View plots of the spectrum analysis for the example sequence:

   • frequency spectrum of aspct (20 coefficients)
   • aspct with sinusoid of fitted frequency and phase

7. Phase values are arbitrary and depend on the point in the gait at which the image sequence begins. We select a single signal to be a reference for the others. Phases, with the reference phase subtracted, are features.

8. Selected features combine to form a feature vector for recognition. The plot shows the average feature values and the maximum and minimum values for each subject in our experiment.
   • feature vector plot