PyCAPIO - Transparent Streaming in File-Based Python Workflows

This work introduces PyCAPIO, a new Python module that transparently improves file-based Python workflow performance by avoiding slow disk I/O and enabling workflow steps to stream data in memory. PyCAPIO interfaces the Python interpreter directly with the CAPIO middleware, eliminating the need for syscall interception. This simplifies the integration of streaming capabilities into Python workflows and provides native bindings for the CAPIO-CL coordination language, making the CAPIO methodology easier to adopt. In addition, removing syscall interception enables support for RISC-V, ARM, and potentially any Linux-based CPU architecture that supports Python. Using both synthetic benchmarks and two real-world workflows (a scientific workflow and an AI-focused workflow), we show that PyCAPIO delivers performance comparable to, or better than, the existing CAPIO approach for Python workflows.