Design and Application of Provenance within Distributed Workflow Management Systems

The growing adoption of computational workflows in scientific research requires robust mechanisms for recording and managing provenance information. Distributed and federated Workflow Management Systems complicate provenance capture: execution is divided across different sites, intermediate states are transient, and data are staged and restaged throughout the workflow execution. Building upon the RO-Crate ecosystem--which packages research outputs with their metadata to make them easier to share, reuse, and understand--we extended the Provenance Run Crate to support distributed WMSs contexts by designing the Distributed Step Crate, a per-step provenance fragment encoded within the distributed node itself. These fragments are collated within the Distributed Provenance Crate, resulting in a contiguous provenance record. We further propose an Orchestration Crate to contain multiple workflow provenance records for a combined view of provenance across complex, distributed workflows. The Distributed Provenance Crate is demonstrated by its application to Globus Flows, a distributed workflow engine. The resulting provenance reproduces the step-level semantics of a reference, centralized Common Workflow Language Provenance Run Crate record, while extending this record with runtime, hardware, and access-control metadata. Finally, we discuss the conditions for the application of the Distributed Step Crate to production distributed WMSs. An open-source reference implementation and example RO-Crates accompany this paper. Together, these results illustrate how provenance can be effectively extended to distributed WMS contexts, enhancing transparency, reproducibility, and collaboration in scientific research.