From: James Kohl <kohlja@ornl.gov>

Date: Wed Sep 10, 2003 3:32:44 PM US/Pacific

To: John Shalf <jshalf@lbl.gov>

Cc: diva@lbl.gov

Subject: Re: DiVA Survey (Please return by Sept 10!)

Reply-To: kohlja@ornl.gov

O.K., here goes... wahoo... :)

=============The Survey=========================

1) Data Structures/Representations/Management==================

The center of every successful modular visualization architecture has

been a flexible core set of data structures for representing data that

is important to the targeted application domain. Before we can begin

working on algorithms, we must come to some agreement on common methods

(either data structures or accessors/method calls) for exchanging data

between components of our vis framework.

There are two potentially disparate motivations for defining the data

representation requirements. In the coarse-grained case, we need to

define standards for exchanging data between components in this

framework (interoperability). In the fined-grained case, we want to

define some canonical data structures that can be used within a

component -- one developed specifically for this framework. These two

use-cases may drive different set of requirements and implementation

issues.

* Do you feel both of these use cases are equally important or

should we focus exclusively on one or the other?

I think for now we need to exclusively focus on exchanging data between

components, rather than any fine-grained generalized data objects...

The first order entry into any component development is to "wrap up

what ya got". The "rip things apart" phase comes after you can glue

all the coarse-grained piece together reliably...

* Do you feel the requirements for each of these use-cases are

aligned or will they involve two separate development tracks?

Two separate development tracks. Definitely. There are different driving

design forces and they can be developed (somewhat) independently (I hope).

For instance, using "accessors" (method calls that provide abstract

access to essentially opaque data structures) will likely work fine for

the coarse-grained data exchanges between components, but will lead to

inefficiencies if used to implement algorithms within a particular

component.

* As you answer the "implementation and requirements" questions

below, please try to identify where coarse-grained and fine-grained use

cases will affect the implementation requirements.

What are requirements for the data representations that must be

supported by a common infrastructure. We will start by answering Pat's

questions of about representation requirements and follow up with

personal experiences involving particular domain scientist's

requirements.

Must: support for structured data

Must.

Must/Want: support for multi-block data?

Must.

Must/Want: support for various unstructured data representations?

(which ones?)

Must/Want: support for adaptive grid standards? Please be specific

about which adaptive grid methods you are referring to. Restricted

block-structured AMR (aligned grids), general block-structured AMR

(rotated grids), hierarchical unstructured AMR, or non-hierarchical

adaptive structured/unstructured meshes.

Must/Want: "vertex-centered" data, "cell-centered" data?

other-centered?

All of these should be "Wants", to the extent that they require more

sophisticated handling, or are less well-known in terms of generalizing

the interfaces.

For example, the AMR folks havfe been trying to get together and define

a standard API, and have been as yet unsuccessful. Who are we to attempt

this where they have failed...?

So to clarify, if we *really* understand (or think we do) a particular

data representation/organization, or even a specific subset of a general

representation type, then by all means lets whittle an API into our stuff.

Otherwise, leave it alone for someone else to do, or do as strictly needed.

Must: support time-varying data, sequenced, streamed data?

MUST.

Must/Want: higher-order elements?

Must/Want: Expression of material interface boundaries and other

special-treatment of boundary conditions.

Wants, see above...

* For commonly understood datatypes like structured and

unstructured, please focus on any features that are commonly overlooked in

typical implementations. For example, often data-centering is overlooked

in structured data representations in vis systems and FEM researchers

commonly criticize vis people for co-mingling geometry with topology

for unstructured grid representations. Few datastructures provide

proper treatment of boundary conditions or material interfaces. Please

describe your personal experience on these matters.

* Please describe data representation requirements for novel data

representations such as bioinformatics and terrestrial sensor datasets.

In particular, how should we handle more abstract data that is

typically given the moniker "information visualization".

I don't think we should "pee in this pool" either yet. Are any of us

experts in this kind of viz? Let's stick with what we collectively know

best and make that work before we try to tackle a related-but-fundamentally-

different-domain.

What do you consider the most elegant/comprehensive implementation for

data representations that you believe could form the basis for a

comprehensive visualization framework?

Sounds like the "Holy Grail" to me... If anything even remotely close to

this already existed, we'd all be using it already...

(Unless of course it's the dreaded NIH syndrome...)

* For instance, AVS uses entirely different datastructures for

structure, unstructured and geometry data. VTK uses class inheritance

to express the similarities between related structures. Ensight treats

unstructured data and geometry nearly interchangably. OpenDX uses more

vector-bundle-like constructs to provide a more unified view of

disparate data structures. FM uses data-accessors (essentially keeping

the data structures opaque).

* Are there any of the requirements above that are not covered by

the structure you propose?

* This should focus on the elegance/usefulness of the core

design-pattern employed by the implementation rather than a

point-by-point description of the implemenation!

* Is there information or characteristics of particular file format

standards that must percolate up into the specific implementation of

the in-memory data structures?

I dunno, but what does HDF5 or NetCDF include? We should definitely be

able to handle various meta-data...

Otherwise, our viz framework should be able to read in all sorts of

file-based data as input, converting it seamlessly into our "Holy Data

Grail" format for all the components to use and pass around. But the

data shouldn't be identifiable as having once been HDF or NetCDF, etc...

(i.e. it's important to read the data format, but not to use it internally)

For the purpose of this survey, "data analysis" is defined broadly as

all non-visual data processing done *after* the simulation code has

finished and *before* "visual analysis".

* Is there a clear dividing line between "data analysis" and "visual

analysis" requirements?

NO. There shouldn't be - these operations are tightly coupled, or even

symbiotic, and *should* all be incorporated into the same framework,

indistinguishable from each other.

* Can we (should we) incorporate data analysis functionality into

this framework, or is it just focused on visual analysis.

YES.

* What kinds of data analysis typically needs to be done in your

field?

Simple sampling, basic statistical averages/deviations, principal component

analysis (PCA, or EOF for climate folks), other dimension reduction.

Please give examples and how these functions are currently

implemented.

C/C++ code... mostly slow serial... :-Q

* How do we incorporate powerful data analysis functionality into

the framework?

As components (duh)... :-)

We should define some "standard" APIs for the desired analysis functions,

and then either wrap existing codes as components or shoehorn in existing

component implementations from systems like ASPECT.

2) Execution Model=======================

It will be necessary for us to agree on a common execution semantics

for our components. Otherwise, while we might have compatible data

structures but incompatible execution requirements. Execution

semantics is akin to the function of protocol in the context of network

serialization of data structures. The motivating questions are as

follows;

* How is the execution model affected by the kinds of

algorithms/system-behaviors we want to implement.

Directly. There are probably a few main exec models we want to cover.

I don't think the list is *that* long...

As such, we should anticipate building several distinct framework

environments that each exclusively support a given exec model. Then

the trick is to "glue" these individual frameworks together so they can

interoperate (exchange data and invoke each others' component methods)

and be arbitrarily "bridged" together to form complex higher-level

pipelines or other local/remote topologies.

* How then will a given execution model affect data structure

implementations

I don't think it should affect the data structure impls at all, per se.

Clearly, the access patterns will be different for various execution models,

but this shouldn't change the data impl. Perhaps a better question is

how to indicate the expected access pattern to allow a given data impl

to optimize or properly prefetch/cache the accesses...

* How will the execution model be translated into execution

semantics on the component level. For example will we need to implement

special control-ports on our components to implement particular execution

models or will the semantics be implicit in the way we structure the

method calls between components.

Components should be "dumb" and let other components or the framework invoke

them as needed for a given execution model. The framework dictates the

control flow, not the component. The API shouldn't change.

If you want multi-threaded components, then the framework better support

that, and the API for the component should take the possibility into account.

What kinds of execution models should be supported by the distributed

visualization architecture

* View dependent algorithms? (These were typically quite difficult

to implement for dataflow visualization environments like AVS5).

Want.

* Out-of-core algorithms

Must. This is a necessary evil of "big data". You need some killer

caching infrastructure throughout the pipeline (e.g. like VizCache).

* Progressive update and hierarchical/multiresolution algorithms?

Must.

* Procedural execution from a single thread of control (ie. using an

commandline language like IDL to interactively control an dynamic or

large parallel back-end)

This is not an execution model, it is a command/control interface issue.

You should be able to have a GUI, programmatic control, or scripting to

dictate interactive control (or "steering" as they call it... :-). The

internal software organization shouldn't change, just the interface to

the outside (or inside) world...

* Dataflow execution models?

Must.

What is the firing method that should

be employed for a dataflow pipeline? Do you need a central executive like

AVS/OpenDX or, completely distributed firing mechanism like that of

VTK, or some sort of abstraction that allows the modules to be used

with either executive paradigm?

This should be an implementation issue in the "dataflow framework", and

should not affect the component-level APIs.

* Support for novel data layouts like space-filling curves?

Must. But this isn't an execution model either. It's a data structure

or algorithmic detail...

* Are there special considerations for collaborative applications?

Surely. The interoperability of distinct framework implementations

ties in with this... but the components shouldn't be aware that they

are being run collaboratively/remotely... definitely a framework issue.

* What else?

Yeah right.

How will the execution model affect our implementation of data

structures?

It shouldn't. The execution model should be kept independent of the

data structures as much as possible.

If you want to build higher-level APIs for specific data access patterns

that's fine, but keep the underlying data consistent where possible.

* how do you decompose a data structure such that it is amenable to

streaming in small chunks?

This sounds a lot like distributed data decompositions. I suspect that

given a desired block/cycle size, you can organize/decompose data in all

sorts of useful ways, depending on the expected access pattern.

In conjunction with this, you could also reorganize static datasets

into filesystem databases, with appropriate naming conventions or

perhaps a special protocol for lining up the data blob files in the

desired order for streaming (in either time or space along any axis).

Meta-data in the files might be handy here, too, if it's indexed

efficiently for fast lookup/searching/selection.

* how do you represent temporal dependencies in that model?

Meta-data, or file naming conventions...

* how do you minimize recomputation in order to regenerate data for

view-dependent algorithms.

No clue.

What are the execution semantics necessary to implement these execution

models?

* how does a component know when to compute new data? (what is the

firing rule)

There are really only 2 possibilities I can see - either a component is

directly invoked by another component or the framework, or else a method

must be triggered by some sort of dataflow dependency or stream-based

event mechanism.

* does coordination of the component execution require a central

executive or can it be implemented using only rules that are local to a

particular component.

This is a framework implementation detail. No. No. Bad Dog.

The component doesn't know what's outside of it (in the rest of the

framework, or the outside world). It only gets invoked, one way or

another.

* how elegantly can execution models be supported by the proposed

execution semantics? Are there some things, like loops or

back-propagation of information that are difficult to implement using a

particular execution semantics?

We need to keep the different execution models separate, as implementation

details of individual frameworks. This separates the concerns here.

How will security considerations affect the execution model?

Ha ha ha ha...

They won't right away, except in collaboration scenarios.

Think "One MPI Per Framework" and do things the old fashioned way

locally, then do the "glue" for inter-framework connectivity with

proper authentication only as needed. (No worse than Globus... :-)

3) Parallelism and load-balancing=================

Thus far, managing parallelism in visualization systems has been a

tedious and difficult at best. Part of this is a lack of powerful

abstractions for managing data-parallelism, load-balancing and

component control.

Please describe the kinds of parallel execution models that must be

supported by a visualization component architecture.

* data-parallel/dataflow pipelines?

Must.

* master/slave work-queues?

Must.

* streaming update for management of pipeline parallelism?

Must.

* chunking mechanisms where the number of chunks may be different

from the number of CPU's employed to process those chunks?

This sounds the same as master/slave to me, as in "bag of tasks"...

* how should one manage parallelism for interactive scripting

languages that have a single thread of control? (eg. I'm using a

commandline language like IDL that interactively drives an arbitrarily

large set of parallel resources. How can I make the parallel back-end

available to a single-threaded interactive thread of control?)

Broadcast, Baby... Either you blast the commands out to everyone SIMD

style (unlikely) or else you talk to the Rank 0 task and the command

gets forwarded on a fast internal network.

Please describe your vision of what kinds of software support /

programming design patterns are needed to better support parallelism

and load balancing.

* What programming model should be employed to express parallelism.

(UPC, MPI, SMP/OpenMP, custom sockets?)

All but UPC will be necessary for various functionality.

* Can you give some examples of frameworks or design patterns that

you consider very promising for support of parallelism and load balancing.

(ie. PNNL Global Arrays or Sandia's Zoltan)

http://www.cs.sandia.gov/Zoltan/

http://www.emsl.pnl.gov/docs/global/ga.html

Nope, that covers my list of hopefuls.

* Should we use novel software abstractions for expressing

parallelism or should the implementation of parallelism simply be an

opaque property of the component? (ie. should there be an abstract

messaging layer or not)

It's not our job to develop "novel" parallelism abstractions. We should

just use existing abstractions like what the CCA is developing.

* How does the NxM work fit in to all of this? Is it sufficiently

differentiated from Zoltan's capabilities?

I don't know what Zoltan can do specifically, but MxN is designed for

basic "parallel data redistribution". This means it is good for doing

big parallel-to-parallel data movement/transformations among two disparate

parallel frameworks, or between two parallel components in the same

framework with different data decompositions. MxN is also good for

"self-transpose" or other types of local data reorganization within a

given (parallel) component.

MxN doesn't do interpolation in space or time (yet, probably for a while),

and it won't wash your car (but it won't drink your beer either... :-).

If you need something fancier, or if you don't really need any data

reorganization between the source and destination of a transfer, then

MxN *isn't* for you...

===============End of Mandatory Section (the rest is

voluntary)=============

4) Graphics and Rendering=================

What do you use for converting geometry and data into images (the

rendering-engine). Please comment on any/all of the following.

* Should we build modules around declarative/streaming methods for

rendering geometry like OpenGL, Chromium and DirectX or should we move

to higher-level representations for graphics offered by scene graphs?

What are the pitfalls of building our component architecture around

scene graphs?

* What about Postscript, PDF and other scale-free output methods for

publication quality graphics? Are pixmaps sufficient?

In a distributed environment, we need to create a rendering subsystem

that can flexibly switch between drawing to a client application by

sending images, sending geometry, or sending geometry fragments

(image-based rendering)? How do we do that?

I would think this could be achieved by a sophisticated data communication

protocol - one that encodes the type of data in the stream, say, using XML

or some such thingy.

* Please describe some rendering models that you would like to see

supported (ie. view-dependent update, progressive update) and how they

would adjust dynamically do changing objective functions (optimize for

fastest framerate, or fastest update on geometry change, or varying

workloads and resource constraints).

* Are there any good examples of such a system?

What is the role of non-polygonal methods for rendering (ie. shaders)?

* Are you using any of the latest gaming features of commodity cards

in your visualization systems today?

* Do you see this changing in the future? (how?)

5) Presentation=========================

It will be necessary to separate the visualization back-end from the

presentation interface. For instance, you may want to have the same

back-end driven by entirely different control-panels/GUIs and displayed

in different display devices (a CAVE vs. a desktop machine). Such

separation is also useful when you want to provide different

implementations of the user-interface depending on the targeted user

community. For instance, visualization experts might desire a

dataflow-like interface for composing visualization workflows whereas a

scientists might desire a domain-specific dash-board like interface

that implements a specific workflow. Both users should be able to

share the same back-end components and implementation even though the

user interface differs considerably.

How do different presentation devices affect the component model?

Not. The display device only affects resolution or bandwidth required.

This could be parameterized in the component invocations APIs, but

should not otherwise change an individual component.

If you want a "multiplexer" to share a massive data stream with a powerwall

and a PDA, then the "multiplexer component" implementation handles that...

* Do different display devices require completely different user

interface paradigms?

No. Different GUIs should all map to some common framework command/control

interface. The same functions will ultimately get executed, just from buttons

with different labels or appl-specific short-cuts... The UIs should all be

independent, but talk the same protocol to the framework.

If so, then we must define a clear separation

between the GUI description and the components performing the back-end

computations. If not, then is there a common language to describe user

interfaces that can be used across platforms?

Yuk.

* Do different display modalities require completely different

component/algorithm implementations for the back-end compute engine?

(what do we do about that??)

Algorithm maybe, component no. This could fall into the venue of the

different execution-model-specific frameworks and/or their bridging...

I dunno.

What Presentation modalities do you feel are important and what do you

consider the most important.

* Desktop graphics (native applications on Windows, on Macs)

MUST.

* Graphics access via Virtual Machines like Java?

Ha ha ha ha...

* CAVEs, Immersadesks, and other VR devices

Must.

* Ultra-high-res/Tiled display devices?

Must.

* Web-based applications?

Probably a good idea. Someone always asks for this... :-Q

What abstractions do you think should be employed to separate the

presentation interface from the back-end compute engine?

Some sort of general protocol descriptor, like XML...? Nuthin fancy.

* Should we be using CCA to define the communication between GUI and

compute engine or should we be using software infrastructure that was

designed specifically for that space? (ie. WSDL, OGSA, or CORBA?)

The CCA doesn't do such communication per se. Messaging between or in/out

of frameworks is always "out of band" relative to CCA port invocations.

If the specific framework impl wants to shove out data on some wire,

then it's hidden below the API level...

I would think that WSDL/SOAP would be O.K. for low-bandwidth uses.

* How do such control interfaces work with parallel applications?

Should the parallel application have a single process that manages the

control interface and broadcasts to all nodes or should the control

interface treat all application processes within a given component as

peers?

I vote for the "single process that manages the control interface and

broadcasts to all nodes" (or the variation above, where one of the

parallel tasks forwards to the rest internally :-). The latter is

not scalable.

BTW, you can't have "application processes within a... component".

What does that even mean?

Usually, an application "process" consists of a collection of one or

more components that have been composed with some specific connectivity...

6) Basic Deployment/Development Environment Issues============

One of the goals of the distributed visualization architecture is

seamless operation on the Grid -- distributed/heterogeneous collections

of machines. However, it is quite difficult to realize such a vision

without some consideration of deployment/portability issues. This

question also touches on issues related to the development environment

and what kinds of development methods should be supported.

What languages do you use for core vis algorithms and frameworks.

C/C++.

* for the numerically intensive parts of vis algorithms

C/C++/Fortran/F90...?

* for the glue that connects your vis algorithms together into an

application?

C/C++.

* How aggressively do you use language-specific features like C++

templates?

RUN AWAYYYY!!! These are not consistent across o.s./arch/compiler yet.

Maybe someday...

* is Fortran important to you? Is it important that a framework

support it seamlessly?

Fortran is crucial for many application scientists. It is not directly

useful for the tools I build.

But if you want to ever integrate application code components directly

into a viz framework, then you better not preclude this... (or Babel...)

* Do you see other languages becoming important for visualization

(ie. Python, UPC, or even BASIC?)

Nope.

What platforms are used for data analysis/visualization?

* What do you and your target users depend on to display results?

(ie. Windows, Linux, SGI, Sun etc..)

All of the above (not so much Sun any more...).

* What kinds of presentation devices are employed (desktops,

portables, handhelds, CAVEs, Access Grids, WebPages/Collaboratories)

and what is their relative importance to active users.

All but handhelds are important, mostly desktops, CAVEs/hi-res and AG,

in decreasing order.

* What is the relative importants of these various presentation

methods from a research standpoint?

CAVEs/hi-res and AG are worthwhile research areas. The rest can be

weaved in or incorporated more easily.

* Do you see other up-and-coming visualization platforms in the

future?

Yes, but I haven't figured out where exactly to stick the chip behind

my ear for the virtual holodeck equipment... :)

Tell us how you deal with the issue of versioning and library

dependencies for software deployment.

CVS.

* For source code distributions, do you bundle builds of all related

libraries with each software release (ie. bundle HDF5 and FLTK source

with each release).

No, but provide web links or separate copies of dependent distributions

next to our software on the web site...

Too ugly to include everything in one big bundle, and not as efficient

as letting the user download just what they need. (As long as everything

you need is centrally located or accessible...)

* What methods are employed to support platform independent builds

(cmake, imake, autoconf). What are the benefits and problems with this

approach.

Mostly autoconf so far. My student thinks automake and libtools is "cool"

but we haven't used them yet...

* For binaries, have you have issues with different versions of

libraries (ie. GLIBC problems on Linux and different JVM

implemetnations/version for Java). Can you tell us about any

sophisticated packaging methods that address some of these problems

(RPM need not apply)

Just say no. Open Source is the way to go, with a small set of "common"

binaries just for yuks. Most times the binaries won't work with the

specific run-time libs anyway...

* How do you handle multiplatform builds?

Autoconf, shared source tree, with arch-specific subdirs for object files,

libs and executables.

How do you (or would you) provide abstractions that hide the locality

of various components of your visualization/data analysis application?

I would use "proxy" components that use out-of-band communication to

forward invocations and data to the actual component implementation.

* Does anyone have ample experience with CORBA, OGSA, DCOM, .NET,

RPC? Please comment on advantages/problems of these technologies.

Nope.

* Do web/grid services come into play here?

Yuk, I hope not.

7) Collaboration ==========================

If you are interested in "collaborative appllications" please define

the term "collaborative". Perhaps provide examples of collaborative

application paradigms.

"Collaborative" is 2 or more geographically/remote teams, sharing one

common viz environment, with shared control and full telepresence.

(Note: by this definition, "collaborative" does not yet exist... :-)

Is collaboration a feature that exists at an application level or are

there key requirements for collaborative applications that necessitate

component-level support?

Collaboration should exist *above* the application level, either outside

the specific framework or as part of the framework "bridging" technology.

* Should collaborative infrastructure be incorporated as a core

feature of very component?

NO. Let the framework proxy to collaborative capabilities...

* Can any conceivable collaborative requirement be satisfied using a

separate set of modules that specifically manage distribution of events

and data in collaborative applications?

I dunno, I doubt it.

* How is the collaborative application presented? Does the

application only need to be collaborative sometimes?

Yes, collaboration should be flexible and on demand as needed - like

dialing out on the speakerphone while in the middle of a meeting...

* Where does performance come in to play? Does the visualization

system or underlying libraries need to be performance-aware?

There likely will need to be "hooks" to specify performance requirements,

like "quality of service". This should perhaps be incorporated as part

of the individual component APIs, or at least metered by the frameworks...

(i.e. I'm

doing a given task and I need a framerate of X for it to be useful

using my current compute resources)...

It would be wise to specify the frame rate requirement, perhaps interactively

depending on the venue... e.g. in interactive collaboration scenarios you'd

rather drop some frames consistently than stall completely or in bursts...

network aware (i.e. the system is

starving for data and must respond by adding an alternate stream or

redeploying the pipeline).

This sounds like futureware to me - an intelligent network protocol layer...

beyond our scope for sure!

Are these considerations implemented at the

component level, framework level, or are they entirely out-of-scope for

our consideration?

These issues should be dealt with mostly at the framework level, if at all.

I think they're mostly out-of-scope for the first incarnation...

WHEW! DONE! :-D

Jeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeembo