The SC00 Application Bandwidth Challenge

The SC conference on high-performance networking and computing has long been a place where high-performance computers and high-speed networks meet. At SC'2000 in Dallas, the Scinet team will be creating a particularly exciting network infrastructure that will include multiple multi-gigabit (OC192) links to the show floor and connections to most high-speed national research networks. In addition, the showfloor itself will feature a state-of-the-art wireless network, as part of the Escape event. We hereby challenge the community to show that this unique network can be used to demonstrate exciting applications.

To this end, we are soliciting proposals for innovative (especially bandwidth-intensive) application demonstrations. The general idea is that....

Our Application Entry - July 15, 2000

1. Title: Using High-Speed WANs and Network Data Cachees to Enable Remote and Distributed Visualization.

2. Primary Contact: W. Bethel, LBL, ewbethel at lbl dot gov

3. Collaborators:

• W. Bethel, J. Shalf, S. Lau, Visualization Group, LBNL. ewbethel at lbl dot gov, jshalf at lbl.gov, slau at lbl.gov
• D. Gunter, J. Lee, B. Tierney, Data Intensive and Distributed Computing Group, LBNL. dkgunter@lbl.gov, jrlee@lbl.gov, bltierney@lbl.gov
• V. Beckner, Center for Computational Sciences and Engineering, LBNL. vebeckner@lbl.gov
• J. Brandt, D. Evensky, H. Chen, Networking Security and Research, Sandia National Laboratory brandt@ca.sandia.gov, evensky@ca.sandia.gov, hycsw@ca.sandia.gov
• G. Pavel, J. Olsen, B. H. Bodtker, Advanced Communications and Signal Processing Group, Electronics Engineering Technologies Division, Lawrence Livermore National Laboratory. pavel1@llnl.gov, olsen11@llnl.gov, bodtker1@llnl.gov

4. Project Description

• a. A paragraph summary of your application and its significance.
  Visapult is a prototype application and framework for remote visualization of large scientific datasets. We approach the technical challenges of tera-scale visualization with a unique architecture that employs high speed WANs and network data caches for data staging and transmission. High throughput rates are achieved by parallelizing i/o at each stage in the application, and by pipelining the visualization process. Visapult consists of two components; a viewer and a back end. The back end is a parallel application that loads in large scientific data sets using a domain decomposition, and performs software volume rendering on each subdomain, producing an image. The viewer, also a parallel application, implements Image Based Rendering Assisted Volume Rendering using the imagery produced by the back end.
  On the display device, graphics interactivity is effectively decoupled from the latency inherent in network applications. This submission seeks to make two high-water marks: remote visualization of a dataset that exceeds 1TB in size, and application performance that will exceed 1.5Gb/s in sustained network bandwidth consumption.
• b. A paragraph summary of hardware, software, and networking resources required for your application.
  The software for the demonstration is the product of a research project, and already exists. Source may be downloaded from the URL (above). This demonstration will require 1.5TB of network storage in the form of multiple DPSS storage caches. By SC00, we expect a DPSS of such capacity and equipped with dual gigabit ethernet interfaces to be online and connected to NTON via an OC-48 link. In order to meet our target of >1.5Gb/s sustained transfer, we require a full OC-48 connection from the data source to the computational engine (Visapult back end). The computational engine performs domain-decomposed software volume rendering, and we require approximately 64 compute nodes to perform rendering at a sustained rate of >1.5Gb/s. The first choice would be a large SMP equipped with multiple gigabit-capable interface cards. Alternately, a large cluster with dual interfaces (one for IPC, the second for external network) would be feasible. In either configuration, the compute engine must be connected to accomodate OC-48 bandwidth. Without an OC-48 connection between the data source and the computational engine, the application will still function, albeit at a slower rate.
• c. A paragraph summarizing how you propose to communicate how your network uses Scinet resources
  The application uses NetLogger, which allows for real time profiling of distributed application performance. We will be able to show NetLogger running simultaneously with Visapult to demonstrate rel-time performance over Scinet.
• d. URL address for further project documentation.
  /projects/visapult/
• e. Graphics illustrating your application are encouraged.
  Visit URL in 4.d.

5. Detailed technical requirements, including answers to these questions:

• a. What capacity network do you expect to require?
  We envision two possible application configurations. In the first configuration, we will locate and use resources connected directly to NTON. Then, we will stream intermediate images used for image based rendering assisted volume rendering to the show floor. In this configuration, we would require an OC-3 link.
  In the second configuration, we would originate data from the DPSS in Berkeley, and use a 64-node system on the show floor for computing the IBR source imagery. In this configuration, we would consume an entire OC-48 link between Berkeley and the show floor. Berkeley has (will have any day now) an OC-48 link to NTON, so we would require an OC-48 link from the show floor to NTON. We anticipate connecting to NTON through HSCC on the show floor.
• b. Where are any off-site resources you will be using? What networks do you expect to use to access them?
  DPSS storage systems at LBL, connected via NTON at OC-48.
• c. In what booth are any on-showfloor resources you will be using?
  LBL - display devices (taxonomy not yet known at this point). The viewer portion of the application is capable of running on any platform that supports OpenGL, X and TCP/IP networking (not yet tested on Win32 - sorry).
  We would be interested in obtaining more information about computational resources that might be available on the show floor.
  We will need connectivity to HSCC on the show floor. This will provide us with access to NTON.
• d. Do you require multicast? If so, to what sites will you connect?
  No multicast required for this application.
• e. Do you require other than IP communications? If so, provide details.
  No, this is all TCP/IP.
• f. Do you require specialized on-showfloor equipment for your demonstration, such as storage, computers, display devices? If so, provide details.
  If a 64 node system (SMP) with multiple gigabit ethernet connections becomes available, we would love to use it.

Current Issues and Planning

If everything in the following lists of items is taken care of, the app and all supporting infrastructure will be in place and ready for execution in the competition. If any are missing or non-functional, we will not be able to participate in the competition.

Raw Data

• One terabyte of data. (Vince Beckner). Nominally, we could use 256 time steps of data from 1024x1024x1024 grid (4 GB per time step) to meet the 1TB target.
• Move data onto the LBL DPSS. (Wes Bethel).

LBL DPSS

• Support for large files. Dan Gunter & Jason Lee. An existing test case (44GB) causes a libdb metadata table overflow.
• Dual channel gigE out of the DPSS. Jason Lee & Dan Gunter?

Cluster/Computing Resources for Visapult Back End

We need to locate a cluster (32 nodes minimum, 64 nodes on an SMP preferred) for use as the Visapult back end. This machine must be connected via a network path capable of sustaining OC-48 speeds while the application is in use. At this time, the best possibility is the CPlant at SNL. Any additional suggestions would be welcome. (Wes Bethel)

On CPlant, we will need a minimum of 32 nodes, though 64 would be better. (Helen Chen and Jim Brandt)

LBL Networking Issues

In order to use the LBL DPSS, we need impediment-free OC-48 connectivity from LBL to the outside world. (Who is responsible for this?)

NTON

If we assume LBL-DPSS and CPlant as the data source and computational resource, we will need access to NTON at OC-48 rates for testing and application execution. (George Pavel)