11-04-2011, 02:35 PM
[attachment=12015]
Grid Computing
• “Flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resources”
• I. Foster, C. Kesselman, S. Tuecke, The Anatomy of the Grid: Enabling Scalable Virtual Organizations, International J. Supercomputer Applications, 15(3), 2001
• Globus, Condor/G, Avaki, EU DataGrid SW, …
• Complexity from User’s Perspective
Process or job model
– Lots of complex state: connections, special shared libraries, licenses, file descriptors
Operating system specificity
– Perhaps even version-specific
– Symbolic supercomputer example
Need to buy into some “Grid API”
• Install and learn complex Grid software
• Complexity from Resource Owner’s Perspective
• Install and learn complex Grid software
• Deal with local accounts and privileges
– Associated with global accounts or certificates
Protection
• Support users with different OS, library, license, etc, needs.
• Virtual Machines
• Language-oriented VMs
– Abstract interpreted machine, JIT Compiler, large library
– Examples: UCSD p-system, Java VM, .NET VM
• Application-oriented VMs
– Redirect library calls to appropriate place
– Examples: Entropia VM
Virtual servers
– Kernel makes it appear that a group of processes are running on a separate instance of the kernel
– Examples: Ensim, Virtuozzo, SODA, …
Virtual machine monitors (VMMs)
– Raw machine is the abstraction
– VM represented by a single image
– Examples: IBM’s VM, VMWare, Virtual PC/Server, Plex/86, SIMICS, Hypervisor, DesQView/TaskView. VM/386
– VMWare GSX VM
• Isn’t It Going to Be Too Slow?
Virtualized NICs have very similar bandwidth, slightly higher latencies
– J. Sugerman, G. Venkitachalam, B-H Lim, “Virtualizing I/O Devices on VMware Workstation’s Hosted Virtual Machine Monitor”, USENIX 2001
• Disk-intensive workloads (kernel build, web service): 30% slowdown
– S. King, G. Dunlap, P. Chen, “OS support for Virtual Machines”, USENIX 2003
Virtuoso
• Approach: Lower level of abstraction
– Raw machines, not processes
• Mechanism: Virtual machine monitors
• Our Focus: Middleware support to hide complexity
– Ordering, instantiation, migration of machines
– Virtual networking and remote devices
– Connectivity to remote files, machines
– Information services
– Monitoring and prediction
– Resource control
The Virtuoso Model
1. User orders raw machine(s)
• Specifies hardware and performance
• Basic software installation available
• OS, libraries, licenses, etc.
2. Virtuoso creates raw image and returns reference
• Image contains disk, memory, configuration, etc.
3. User “powers up” machine
4. Virtuoso chooses provider
• Information service
5. Virtuoso migrates image to provider
• Efficient network transfer
• rsync, demand paging, versioned filesystems
6. Provider instantiates machine
• Virtual networking ties machine back to user’s home network
• Remote device support makes user’s desktop’s devices available on remote VM
• Remote display support gives user the console of the machine (VNC)
• Resource control to give user expected performance
7. User goes to his network admin to get address, routing for his new machine
8. User customizes machine
• Feeds in CDs, floppies, ftp, up2date, etc.
9. User uses machine
• Shutdown, hibernate, power-off, throw away
10. Virtuoso continuously monitors and adapts
• Various mechanisms, all invisible to user
• Migrating the machine
• Routing traffic between machines
• Virtual network topology
• Predictive scheduling versus reservations
• Various goals
• Price
• Interactivity
• Information service
• Resource monitoring and prediction
Why Virtual Networking?
• A machine is suddenly plugged into your network. What happens?
– Does it get an IP address?
– Is it a routeable address?
– Does firewall let its traffic through?
– To any port?
• A Layer 2 Virtual Network (VLAN) for the User’s Virtual Machines
Why Layer 2?
– Protocol agnostic
– Mobility
– Simple to understand
– Ubiquity of Ethernet on end-systems
What about scaling?
– Number of VMs limited
– Hierarchical routing possible because MAC addresses can be assigned hierarchically
– A Simple Layer 2 Virtual Network
• A Simple Layer 2 Virtual Network
• A Simple Layer 2 Virtual Network
An Overlay Network
• Bridgeds and connections form an overlay network for routing traffic among virtual machines and the user’s home network
• Links can trivially be added or removed
Bootstrapping the Virtual Network
• Star topology always possible
• TCP session from client must have been possible
• Better topology may be possible
• Depends on security at each site
• Topology may change
• Virtual machines can migrate
• Bootstrap to higher layers
• Virtual filesystems
Remote Devices
• Extending a Grid Information Service (GIS) to Support Virtual Machines
• A GIS contains information about the available resources in a grid
– Hosts, routers, switches, software, etc.
• URGIS project at Northwestern
– GIS based on the relational data model
– Compositional queries (joins) to find collections of resources.
• “Find physical machines which can instantiate a virtual machine with 1 GB of memory”
• “Find sets of four different virtual machines on the same network with a total memory between 512 MB and 1 GB”
– Nondeterministic query extension for scalability
• http://cs.northwestern.edu/~urgis
• The RGIS Design (Per Site)
• Deadlines
• Extending a Grid Information Service (GIS) to Support Virtual Machines
Virtual indirection
– Each RGIS object has a unique id
– Virtualization table associates unique id of virtual resources with unique ids of their constituent physical resources
– Virtual nature of resource is hidden unless query explicitly requests it
Futures
– An RGIS object that does not exist yet
– Futures table of unique ids
– Future nature of resource hidden unless query explicitly requests it
• Extending a Resource Monitoring and Prediction System to Support Virtual Machines
• Measuring and predicting dynamic resource availability to support adaptation
– Virtual machine migration
– Routing on the virtual network
– Application-level adaptation
RPS System at Northwestern
– Host and network measurements for Unix and Windows
– Emphasis on prediction (wide range of linear and nonlinear models) and communication (wide range of transports)
RPS Toolkit
• Extensible toolkit for implementing resource signal prediction systems [CMU-CS-99-138]
• Growing: RTA, RTSA, Wavelets, GUI, etc
• Easy “buy-in” for users
• C++ and sockets (no threads)
• Prebuilt prediction components
• Libraries (sensors, time series, communication)
