Rapid Prototyping of Parallel Robot Vision Systems Using Virtual Reality and Systems Simulation

Progress Report, 1996-97

Randal Nelson, Dana Ballard, Chris Brown,
Thomas LeBlanc, and Michael Scott
Department of Computer Science
University of Rochester
NSF CDA-9401142

Introduction

Institution

The University of Rochester is a small, private University established in 1850, originally as a Baptist-sponsored institution. During the early 20th century, the University grew significantly, in part due to the efforts of George Eastman, the founder of Eastman Kodak. During this period, the Medical School, the Institute of Optics, and the Eastman School of Music, all currently nationally known, were established. Today, the University is home to 4600 undergraduates and 2500 graduate students, and operates with a philosophy of providing the academic opportunities of a renowned research institution in an environment scaled to the individual. The Department of Computer Science at the University of Rochester offers an intense, research-oriented program leading to the degree of Doctor of Philosophy, with particular emphasis on the areas of computer vision and robotics, knowledge representation and natural language understanding, systems software for parallel computing, and the theory of computation. The focused research interests reflect our desire to achieve excellence in a core of important issues, rather than to try to cover all areas.

Project

This document is a third-year progress report for NSF Institutional Infrastructure Grant No. CDA-9401142, ``Rapid Prototyping of Parallel Robot Vision Systems Using Virtual Reality and Systems Simulation.'' (See main project web page.) The main focus of our research is a laboratory that combines sensory interaction with physical environments (otherwise known as virtual reality) with execution-driven simulation of complex parallel systems in the design of visually-controlled robotic systems. This research leverages Rochester's unique combination of expertise in active vision systems, behavioral robotics, virtual reality, and parallel programming environments and systems. Our laboratory has two halves: one half is for building working systems in the real world; the other half is for prototyping and experimentation in the virtual world. The components of the real-world laboratory are the effectors and sensors for interacting in the real world, and the computational machinery required to run the control algorithms. The virtual-world laboratory includes the hardware and software for creating the virtual world, including models of sensors, effectors, and their interaction with an environment.

Year's Activities

Goals, Objectives and Targeted Activities

Overall, our goal was to continue to develop the dual real/virtual world prototyping environment and hardware infrastructure, while using the structure already in place to carry out research in various areas of parallel visual perception and control. On the real-world side, our goals focussed on techniques to eliminate the need for detailed prior models in robot control and visual perception. These included (1) continuing development of adaptive visual servoing, specification and control; (2) complexity reduction and control for high-degree-of-freedom robot effectors; (3) development and evaluation of adaptive tracking systems; and (4) development of robust, general 3-D object recognizers trainable from purely visual input, and parallelization of the above. In the virtual world a primary objective was (1) closing the loop using real-time visual perception and control algorithms in a realistic simulated world. We also wanted to place emphasis on (2) uncalibrated methods of fusing artificial graphics with real video (augmented reality), and (3) generating and manipulating photo-realistic, non-metric object models from video. Systems support goals were generally (1) to increase our understanding of parallel applications, particularly those involved in simulation and in real-world robotics applications, and (2) to enhance our ability to run these programs efficiently on commodity hardware. Specifically, objectives were (1) to develop compiler optimizations to improve the performance of parallel applications, (2) to develop operating system and run-time techniques to simulate shared memory efficiently on networks of small multiprocessors, and (3) to enhance our performance prediction and analysis tools.

Components and Materials Required

A major resource, acquired in November of 1995 (and described in detail in last year's report), is a 32-processor DEC AlphaServer system connected by DEC's Memory Channel network. The acquisition used RI funds to leverage a major external research grant from DEC. For communication among the platforms dedicated to the RI-funded research, we upgraded the network on our various multiprocessors, graphics engines, and robot controllers this year to 100 Base-T. We also leveraged RI funds with an NIH research center grant to acquire major new capability in graphics engine (SGI infinite reality machine), and haptic sensory and feedback devices (4 phantom virtual reality systems). A memory-loaded multiprocessor (2 gig of core) saw heavy use in the development of the object recognition system. A major purchase of robot manipulators is planned for the near future. On a 50-50 cost-share basis, the grant funds a modest amount (.5 FTE) of technical staff support for specialized robotic and virtual reality equipment. Institutional matching also funds 3 graduate students (9-mo. stipend and tuition). Seven faculty members, 2-3 postdocs and visitors, and approximately 24 graduate students are engaged in research directly related to the equipment provided by the grant, but without financial support from the grant per se.

Indications of Success

Various robotic, virtual reality, and parallel processing resources have been merged into an integrated laboratory for research in intelligent action in real-world environments. The key innovative feature of the laboratory is the ability to switch transparently between real and high-fidelity artificial (simulated) worlds. Projects supported include The parallel processing infrastructure played a key role in attracting a new faculty member, Sandhya Dwarkadas (systems, female). Altogether there were 35-40 refereed publications and about 20 other publications directly related to the infrastructure. 23 Ph.D. students, 2 of them women, made direct use of the infrastructure. 4 finished this year.

Accomplishments

Significant progress was made on most of the goals described above. Due to lack of space, we refer the reader to the references for particulars.

Outcome and Impact

Degree of Success

Overall this was an outstanding year on this grant. We closed the loop on the real vision in a virtual world for the first time, using a photorealistic driving simulator to test real-time algorithms for detection and response to stop-lights and stop signs. We developed uncalibrated methods of generating photorealistic models and embedding video overlay into live video [4, 8, 9] We demonstrated model-free, general 3-D object recognition systems whose performance exceeds the best reported in the literature [12, 13, 14] We also made substantial additional progress on visual manipulation, and control of complex manipulators [2, 3, 5] On the systems side, we made major progress on the fronts of low-cost, high-performance distributed shared memory; performance prediction and analysis; and parallelizing compiler technology [1, 6, 7, 10, 11]

Unmet Goals

The original grant proposal called for running robotic control software on top of a multiprocessor simulator (when running in time-dilated virtual environments), in order to analyze fine-grain performance phenomena such as memory system latencies. This portion of the research has proven to be of lower priority, and is largely subsumed by the capabilities of the Carnival toolkit, which has a dramatically lower impact on program run-times.

Outcome

The grant has helped us retain our position as a leader in active paradigm, biologically-motivated vision and robotics research, and in shared-memory parallel computing. It has helped us recruit and train the very best graduate students (consistently the best in the College on standardized tests), and place them at the best institutions upon their graduation. At the national and global level, work supported by the grant has its impact through the influence of our papers on other researchers, and through the work of our alumni. Our work is heavily cited. Our laboratory techniques (e.g. verging cameras) have been widely copied. We have also been successful at technology transfer. Our Mint simulation tookit is used at scores of sites around the world. Our synchronization algorithms are widely used by other groups, and have inspired a number of follow-on projects. Our clustered Cashmere protocol is the subject of a pending joint patent by the University and Digital Equipment Corp. % Impact (optional) % We are really out of room at this point

Lists

Graduate Students Supported

Maged Michael Galen Hunt Robert Stets Nikolaos Hardavellas
Sotirios Ioannidis Wagner Meira Michal Cierniak Mohammed Zaki
Srinivasan Parthasarathy Martin Jagersand Olac Fuentes Andrea Selinger
Raj Rao Jessica Bayliss Garbis Salgian Rodrigo Carceroni
Jim Vallino Christopher Eveland Mike VanWie

Recent PhD Graduates Supported

Bibliography

  1. M. Cierniak and W. Li. Recovering logical data and code structures. In U. Banerjee, D. Gelernter, A. Nicolau, and D. Padua, editors, Languages and Compilers for Parallel Computing, volume 1033 of Lecture Notes in Computer Science. Springer-Verlag, 1996.
  2. O. Fuentes and R. C. Nelson. The virtual tool approach to dextrous telemanipulation. In Proceedings of the 1996 IEEE International Conference on Robotics and Automation, pages 1700--1705, Minneapolis, Minnesota, April 1996.
  3. O. Fuentes and R. C. Nelson. Learning dextrous manipulation skills using the evolution strategy. In Proceedings of the 1997 IEEE International Conference on Robotics and Automation, Albuquerque, New Mexico, April 1997.
  4. M. Jagersand. Model free view synthesis of an articulated agent. In Proc. Computer Vision and Pattern Recognition, 1997.
  5. M. Jagersand, O. Fuentes, and R. C. Nelson. Experimental evaluation of uncalibrated visual servoing for precision manipulation. In Proceedings of the 1997 IEEE International Conference on Robotics and Automation, Albuquerque, New Mexico, April 1997.
  6. L. Kontothanassis, G. Hunt, R. Stets, N. Hardavellas, M. Cierniak, S. Parthasarathy, W. Meira, S. Dwarkadas, and M. L. Scott. Vm-based shared memory on low-latency, remote-memory-access networks. In Proc. of the 24th Intl. Symp. on Computer Architecture, Denver, CO, June 1997.
  7. L. I. Kontothanassis, R. W. Wisniewski, and M. L. Scott. Scheduler-conscious synchronization. ACM Trans. on Computer Systems, 14(1), Feb. 1997.
  8. K. N. Kutulakos. Shape from the light field boundary. In Proc. Computer Vision and Pattern Recognition, 1997. To appear.
  9. K. N. Kutulakos and J. Vallino. Affine object representations for calibration-free augmented reality. In Proc. IEEE Virtual Reality Annual Symp., pages 25--36, 1996.
  10. W. Meira Jr., T. LeBlanc, and A. Poulos. Waiting time analysis and performance visualization in Carnival. In Proc. of the ACM SIGMETRICS Symp. on Parallel and Distributed Tools (SPDT), pages 1--10, Philadelphia, PA, May 1996.
  11. M. M. Michael and M. L. Scott. Simple, fast, and practical non-blocking and blocking concurrent queue algorithms. In Proc. of the 15th ACM Symp. Principles of Distributed Computing, Philadelphia, PA, May 1996.
  12. R. C. Nelson and A. Selinger. Experiments on (intelligent) brute-force methods for appearance-based object recognition. In Proc. DARPA Image Understanding Workshop, New Orleans, LA, May 1997.
  13. R. Rao. Dynamic appearance-based recognition. In Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 1997.
  14. R. Rao and D. Ballard. Dynamic model of visual recognition predicts neural response properties in the visual cortex. Neural Computation, 9:805--847, 1997.
Last Change: 7 May 1997 / scott@cs.rochester.edu