A codesign framework for online data analysis and reduction

Kshitij Mehta, Bryce Allen, Matthew Wolf, Jeremy Logan, Eric Suchyta, Swati Singhal, Jong Y. Choi, Keichi Takahashi, Kevin Huck, Igor Yakushin, Alan Sussman, Todd Munson, Ian Foster, Scott Klasky

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)

Abstract

Science applications preparing for the exascale era are increasingly exploring in situ computations comprising of simulation-analysis-reduction pipelines coupled in-memory. Efficient composition and execution of such complex pipelines for a target platform is a codesign process that evaluates the impact and tradeoffs of various application- and system-specific parameters. In this article, we describe a toolset for automating performance studies of composed HPC applications that perform online data reduction and analysis. We describe Cheetah, a new framework for composing parametric studies on coupled applications, and Savanna, a runtime engine for orchestrating and executing campaigns of codesign experiments. This toolset facilitates understanding the impact of various factors such as process placement, synchronicity of algorithms, and storage versus compute requirements for online analysis of large data. Ultimately, we aim to create a catalog of performance results that can help scientists understand tradeoffs when designing next-generation simulations that make use of online processing techniques. We illustrate the design of Cheetah and Savanna, and present application examples that use this framework to conduct codesign studies on small clusters as well as leadership class supercomputers.

Original languageEnglish
Article numbere6519
JournalConcurrency Computation Practice and Experience
Volume34
Issue number14
DOIs
Publication statusPublished - 2022 Jun 25

Keywords

  • Cheetah
  • CODAR
  • codesign
  • exascale
  • in situ
  • online
  • reduction
  • Savanna
  • workflows

Fingerprint

Dive into the research topics of 'A codesign framework for online data analysis and reduction'. Together they form a unique fingerprint.

Cite this