Abstract: Researchers designing robots that can perform non-trivial behaviors in real-world environments are faced with a problem. There is a growing consensus that machines for several highly desirable applications, especially visually guided navigation and manipulation, must incorporate mechanisms peculiar to their function, and must interact with the world in order to derive the necessary information. Working with sophisticated, physical manipulators or navigators however, is an extremely expensive, time-consuming, and in some ways hazardous process. Furthermore, controlling machinery in the real world requires dealing with real-time interactions among multiple complex computational processes. The consequent risk, complexity, and expense has limited development of non-trivial visually controlled machines to a few well-funded institutions, and even there development has been slow.

There may, however, be a way out. We believe that recent developments in sophisticated simulation technology in two specific areas have crossed a threshold that will allow their use to enormously speed up the design phase for complex, real-world robotic mechanisms. The two technologies are simulation of sensory interaction with physical environments popularly known as virtual reality, and sophisticated simulation of complex parallel algorithms at the level of individual messages and memory accesses. The first technology provides tools that can facilitate the design and testing of the entire software control interface, including low and intermediate level vision, prior to invoking real physical devices and environments. The second provides tools to address the performance and low-level real-time problems of interacting processes. We are currently building a laboratory that will allow us to use these new simulation technologies in the design of visually controlled robots. A primary goal of the project is to produce exportable protocols for specifying simulated development environments. Such protocols will ultimately permit projects in which many institutions can participate in the development of control software for a particular application without necessarily having all the hardware components in house.