DataSpaces is a programming system targeted at current large-scale systems and designed to support dynamic interaction and coordination patterns between scientific applications. DataSpaces essentially provides a semantically specialized shared-space abstraction using a set of staging nodes. This abstraction derives from the tuple-space model and can be associatively accessed by the interacting applications of a simulation workflow. DataSpaces also provides services including distributed in-memory associative object store, scalable messaging, as well as runtime mapping and scheduling of online data analysis operations.
DataSpaces is currently being used by production coupled scientific simulation workflow on large-scale supercomputers. For example, as part of the coupled fusion simulation workflow framework, DataSpaces enables memory-to-memory coupling between the gyrokinetic PIC edge simulation code XGC0, and the MHD code M3D-OMP. Similarly, as part of turbulent combustion workflow DataSpaces enables data coupling between the direct numerical simulations (DNS) code S3D and the data analytics pipeline. DataSpaces has been integrated with and deployed as part of the Adaptive IO System (ADIOS) framework distributed by Oak Ridge National Laboratories (http://www.olcf.ornl.gov/center-projects/adios/). ADIOS is an open source I/O middleware package that has been shown to scale to hundreds of thousands of cores and is being used by a very wide range of applications.
Here is a video of DataSpaces demo with ADIOS/Paraview/PIxel3D
DataSpaces 1.7.0 has been released. This release adds DataSpaces-as-a-Service (DSaaS) persistent server functionality. Addtionally, 1.7.0 provides Hybrid Staging capabilities, and a number of bug fixes and performance improvements. It is available in the Download section.
DataSpaces 1.6.2 has been released with support for Cray Dynamic RDMA credentials (NERSC Cori) and improved support for the IBM XL compiler, among other bug fixes. It is available in the Download section.
We are excited to announce the addition of full-time developer, Philip Davis, to the DataSpaces team.
Qian Sun was awarded the best paper award at the Second International Workshop on Extreme Scale Programming Models and Middleware (ESPM2) at SC16 for her paper entitled, "In-staging data placement for asynchronous coupling of task-based scientific workflows."