NSF Workshop on
New Challenges and Directions for Systems Research
St. Louis, Missouri
August 1, 1997
On the second day of the workshop, approximately 16 people (roughtly 40% of the total workshop attendance) chose to join the ``cross-cutting group'' devoted to programming support and resource management for wide-area applications. To encourage more lively discussion, the group divided arbitrary in half. This page reports discussion in the group composed of
Pei CaoAs shown in the summary slides, discussion focused on four main topics:
John Carter
David Culler
Franz Kaashoek
David Oran
Michael Scott
Liuba Shrira
Willy Zwaenepoel
The group discussed more than a dozen applications. The three that engendered the liveliest discussion were distributed collaboration, distributed access to large simulations (e.g. of the weather), and global file systems. Distributed collaboration is interesting because it can involve both reading and writing of shared data by a potentially large number of users, e.g. for command and control applications. Simulation is interesting because it raises the possibility of steering the computation to be performed based on the portions of the output that users want to see (don't calculate the weather for next Tuesday in central Alaska unless someone explicitly requests it). A shared global file system is interesting because it touches on a wide variety of traditional systems issues: replication and availability, fault tolerance, authentication and security, caching policies, atomicity and coherence, etc. If coherence semantics can be tailored to the needs of the application, then many of the issues discussed in Thursday's distributed state sharing group arise.
It may be useful to distinguish between applications that coordinate concurrent activities by multiple human beings and those that comprise concurrent activities by multiple computational components: timing and quality of service requirements for these two groups may be very different. It may also be useful to distinguish between applications that gather and process data from distributed sources and those that manage computations at distributed sites. Neither distinction is sharp; in both cases there are important applications whose activities are mixed.
Several items in each of the lists on the slides (new problems, new needs, programming support) pertain to resource management. Resources include such low-level notions as cycles, network bandwidth (and maybe latency), and memory and disk space. They may also include requirements, such as not-to-exceed dollar costs or wall-clock deadlines, and constraints, such as the need for a particular processor architecture, operating system, referencing environment, memory or file consistency model, or set of I/O devices. In some cases, it may be appropriate to express requirements in higher-level terms (e.g. polygons/sec. for rendering).
An important obstacle to resource management is the sheer difficulty of figuring out what the resources required by an application are. Imagine, for example, the task of moving a complex system cross-country, from one laboratory to another. One way to do this is to bundle up every single piece of hardware and software and plug it back together in exactly the same way. A more attractive approach is to move only those distinctive components that are unavailable at the other site, and to count on the availability of ``commodity'' infrastructure. But anybody who tries this is likely to encounter an enormous number of unexpected ``gotcha''s -- resource requirements -- that make the task much more difficult than it ``ought'' to be..
Assuming that we can get a handle on the nature of resources, the subgroup agreed that naming and discovery/perusal mechanisms -- which we customarily think of in terms of files, objects, etc. -- need to apply to resources as well. Programs need to be able to specify the resources they need and to hunt for places where those resources might be available. Moreover, in order to reconcile the resources that a program needs with those that a system can provide, interface specifications need to mention resources explicitly. Specifications that describe functionality only will not suffice.
Over time, the resources required by (or available to) an appliaction can be expected to grow and shrink, sometimes suddenly. In some cases an application will want to request additional resources to deal with increased demand (imagine a server for a suddenly-popular web site, which needs to replicate itself). In other cases an application may need to be notified of the loss of resources, so it can modify its behavior (imagine a server that loses some of its bandwidth, and needs to reduce the fidelity of transmitted images to compensate).
Despite the many variations among local operating systems, applications can generally count on a hierarchical file system, virtual memory, multiprogramming, a window system, TCP/IP network access, etc. This degree of consensus does not yet exist in the distributed case. It is clear that the new problems and needs of wide-area applications will call for new programming support. A key goal of future work is to develop a consensus on what that support should look like (and then of course to build it).
--> munging distributed data v. managing distributed
computation
ps, top, ^C)
--> overprovision v. QOS
