Introduction

The DPSS is a network-accessible, "block data" device. Network-accessible means that the DPSS, as a data source, is located "somewhere out on the network." "Block data" device means that the DPSS is intended to be used for large, block data transfers. According to its designers, the DPSS performs best when data is moved in 16MB blocks per request. The API for the DPSS looks like a low-level UNIX file library, so it has routines for things like "file open," "file write," "file read" and "file seek." One of the DPSS claims to fame is that it is an economically scalable storage system - consider a federation of Linux-based PC's, each with a large IDE disk as the DPSS back end, and a front end (a master) that can propogate client data requests from the client to the DPSS workers.

Which DPSS should you use? That's a good question, and there's no easy answer. As of the time of this writing, there are only a couple of experimental DPSS systems in existence. Since these are experimental, there are no claims of production level quality, with features such as security or reliability. This situation may change in time, but for now, you have to hang out with the right people to know which DPSS to use.

DPSS Data Sources and Metadata

Given that the DPSS is a block-level device, the real challenge in making most efficient use of this resource is to collect large amounts of data into large homogeneous blocks, and to transfer as much as possible in as few calls as possible. To this end, there is a "DPSS data format" used by Visapult. This data format is certainly not the most advanced design, but only serves to make the most efficient use of the DPSS by fostering a small number of large payload transfers.

The two components of a Visapult DPSS data set are a "config file" and the accompanying "large data" files. When you run the Visapult back end, and request DPSS as a data source, you will provide the fully-qualified name of this config file (more on running Visapult will come later in this document).

Visapult-DPSS Config File

The config file is a plain old ASCII file that contains information about the size of the problem, what variables are present along with their names, and other information. The format of this file is subject to change without notice. In contrast, the large data files are raw binary IEEE big-endian floats that contain ALL the data for a given species for ALL time steps. More information about these files will follow.

Here's a sample config file, which we'll call "explosion-config":

explosion
640 256 256 10 1
density 1.34088 2.64286 
 

1. Filename base
   The first line of the config file contains a newline-terminated string that is used as the filename base. This string is combined with the species names in order to derive filenames for the large data files.
2. Problem Size
   The second line of the config file contains the width, height, depth, number of time steps and number of variables for this problem. At this time, only one variable is permitted per config file. This will change in the future to allow multiple variables to be present in a single config file.
3. Variable name and min/max
   Each of the following N lines, where N ranges from 1 to the number of variables in the problem, contains a string followed by two floating point values represented as ASCII strings. The first string is the name of the variable, in this case, "density." The two floats represent the minimum and maximum value of the particular variable over ALL time steps. At this time, these values are not used by Visapult, but are helpful in the sense that the Visapult back end provides command line options -min XX -max YY that allow the user to set the min/max values used for the color transfer process used when data is converted to color.
   In the future, it will be possible to specify more than a single variable in a config file, but for now (3/2000), use just one variable per config file.

Visapult Large Data File

The Visapult Back end will read the config file at start up time, and from the config file, construct a file name of the form:

Building a DPSS-Enabled Visapult

1. Grab and Install DPSS Client Libraries
2. Build DPSS-Enabled Visapult

Running a DPSS-Enabled Visapult