Where is Computer Science going as a field? What will the future look like? What role will Rochester play?

We envision a new era of intuitive computing, in which users collaborate in their everyday activities with an intelligent digital assistant that models user intent, and that suggests or carries out actions likely to satisfy that intent. Intuitive computing systems will “belong” to individual users, travel with them, and “understand” their needs. They will actively monitor the physical environment; learn from past experience; seek out and utilize distributed information and computational resources; use innovative algorithms to compute the most promising courses of action in the face of complex, incomplete, or contradictory inputs; and employ powerful yet natural spoken language and graphical interfaces.

In the context of our work the term “intuitive” applies both to user intuition regarding system behavior and to two forms of system “intuition”: inference/prediction of user intent, and selection of promising actions to satisfy that intent. These meanings are mutually supportive, in the sense that the modeling and support of user intent will make systems dramatically easier to use, allowing them to interact with human beings in a way that more satisfyingly resembles the ways in which human beings interact with one another.

Though it may employ resources spread across the Internet, an intuitive computing system will have an interactive locus that travels with the user, much like today's PDAs. It will, however, have multiple physical components, some embedded in clothing, eyeglasses, or even the body. Using multisensory input, including sound and vision, the system will monitor the physical environment and the user's focus of attention. It will then combine this environmental information with past experience, commonsense knowledge, and task-specific knowledge to drive a sophisticated model of user intent and feasibility, allowing it to resolve ambiguous requests, to propose appropriate actions, and to anticipate future needs.

An intuitive computing system will perform a very wide variety of tasks. It will augment human memory, answer questions, and gather and organize information for upcoming activities. It will give its user warnings, reminders, and advice. Both proactively and on demand, it will negotiate with systems associated with other human beings, with organizations, and with objects in the surrounding environment. Like the quintesential butler, it will be instantly at hand when needed, yet completely unobtrusive otherwise.

While many intuitive computing tasks will be short-term and highly constrained, others will be much more open-ended, and capable of consuming arbitrary amounts of computational and other resources. The algorithms to perform these tasks will employ both heuristic and traditionally rigorous techniques, with the choice among them driven by intrinsic problem complexity, resource availability, and the nature (completeness, consistency, accuracy) of the input data. Challenges include choosing the most promising actions in the absence of complete knowledge of external state and consequences, balancing resource requirements against quality of results, allocating available resources among competing goals and obligations, managing and mining large amounts of current and past sensory information, and selecting the order of access to outside information sources.

In an internetworked world, an intuitive computing system will communicate extensively with information sources around the Internet. To maximize its capabilities while conserving energy on local devices, it will offload expensive computations whenever possible. To minimize response time, it will perform this offloading to nearby servers whenever possible. When resources are plentiful, the system will speculatively gather and organize information in anticipation of future needs. When resources are scarce, it will use its knowledge of user focus and its model of user intent to allocate what resources it has to the tasks its computations determine to be the most likely to meet user needs.

Research challenges

Artificial Intelligence, Natural Language, and Knowledge Representation

• planning for the future based on uncertain or incomplete knowledge;
• natural language understanding for multiple speakers in noisy environments;
• modeling of intent in general contexts;
• knowledge acquisition and representation; learning and inference
• intelligent agency: deciding what to do and how to do it, balancing conflicting goals and obligations against each other, and against available resources;
• integration of symbolic and spatial knowledge bases

Computer Vision, Robotics, and Virtual Reality

• object recognition in general environments;
• recognition of spatial location/context and user/human activity;
• connection of visual perception to primitives useful in natural language and planning
• tracking and exploitation of user attention and focus;
• real-time integration of augmented reality with vision and eye tracking

Computer Systems

• generation of secure mobile compoents with embedded resource (computation, communication, data, energy) and quality of service (response time, throughput, accuracy) requirements;
• application partitioning and relocation mechanisms; optimization techniques to distribute these components to maximize performance and minimize cost;
• compile-time prediction and run-time monitoring of data access patterns for locality optimization and usage prediction;
• attention- and intention-driven scheduling and dynamic component offloading;
• naming, discovery, and routing for wide-area agents and resources, using peer-to-peer and mobile ad hoc techniques;
• high-performance, low-energy processing and communications for mobile platforms

Theory of Computation

• algorithmic task decomposition and flexible serialization/ parallelization;
• anticipatory computing via exact and approximation algorithms, and stochastic modeling of human/machine behavior and environments;
• approximation algorithms (including progressive approximations);
• complexity studies to clarify the demarcation line between exactly solvable subtasks and those to be approached approximately or heuristically;
• computing with uncertainty (regarding input, environments, and payoffs);
• cryptographic security;
• data mining/warehousing/classification/clustering;
• query-order theory to improve information access patterns