Abstract: The objective of DARPA IU research at Rochester is to enhance battlefield awareness through the automatic recognition of objects and activities, and then by presenting graphic presentations of them in live video. Several research questions arise: object- and activity- recognition; real-time tracking of natural features and the determination of the state (pose and velocity) of remote objects; the representation of objects for computer processing; the registration of graphic objects with live video; the use of stereo information; the control of autonomous surveillance equipment. We evaluate our work by quantifying how each modification or technique affects sytem performance and by quantifying the reliability, speed, robustness, and versatility of our algorithms.