A Domain-Specific Language for In Situ and In Transit Workflows

2017 SIAM Conference on Computational Science and Engineering

A Domain-Specific Language for In Situ and In Transit Workflows

Abstract. The path to exascale computing is hindered by the ever-widening gap between computational ability and I/O bandwidth. However, as a consequence of increased compute power, the I/O requests and data sizes generated by scientific workflows has grown to unprecedented scales, overwhelming the underlying network and parallel file system and rendering traditional post-processing methods infeasible. One proposed technique aimed at mitigating the I/O bottleneck is operating on the data in situ (i.e., where it is generated) or in transit (i.e., close to where it is generated) during runtime. However, the proper abstractions for describing this type of realtime processing in scientific workflows do not exist. To solve that problem, this work introduces a domain-specific language for users to describe the coupling dynamics and workflow behavior of data-intensive scientific applications. It also defines insights and policies that stand to be gained by knowing more about the overall workflow behavior. Preliminary results show more efficient data placement and an overall reduction in runtime.

Authors