Development of a Python Interface for a Framework of Cross-reference Solar-Terrestrial Physics Simulation

Development of a Python Interface for a Framework of Cross-reference Solar-Terrestrial Physics Simulation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Development of a Python Interface for a Framework of Cross-reference Solar-Terrestrial Physics Simulation Keiichiro FUKAZAWA, Haichao ZHAO, Takeshi NANRI, Yohei MIYAKE, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9195901/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 5 You are reading this latest preprint version Abstract Cross-reference simulation is a calculation model combining multiple simulation codes when some simulation codes need the data in other simulation codes to do their own calculation. To provide an efficient and convenient solution for cross-reference Solar-Terrestrial Physics (STP) simulation, a framework called Code-To-Code Adapter (CoToCoA) Library has been developed. CoToCoA was designed to implement cross-reference simulation with minimal modifications to the simulation codes while maintaining low overhead and currently supports only C and Fortran. To further enhance its usability by integrating CoToCoA with Python, the development of a Python interface is performed. Two development approaches were conducted: (1) direct development of CoToCoA in Python and (2) the use of a library to call C functions of CoToCoA from Python. To evaluate the performance of the Python interfaces, experiments of data transfer methods in CoToCoA are performed. As a result of the evaluation, using the library implementation has a close performance to C, while the direct implementation needs more additional time. The results show that the Python interface using the library enables efficient data transfer from both C and Fortran-based STP simulation programs to Python for advanced data analysis and processing. Furthermore, the developed Python interface was applied to visualize magnetohydrodynamics simulation results in real-time. Based on the application, the framework has shown the potential to enable direct integration with other Python tools, such as machine learning or artificial intelligence models. Coupled Computation Cross-reference Simulation Python Interface Real-time visualization Real-time AI training Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 Figure 12 1 Introduction Coupled computations constitute a fundamental methodology in scientific research, enabling multiple simulation programs to interact collaboratively to address complex problems that exceed the capabilities of individual codes. This approach is particularly significant in the field of Solar-Terrestrial Physics (STP), where diverse physical processes operating at various scales are intrinsically linked and mutually influential. Within coupled computations, each simulation program typically addresses a distinct physical phenomenon and requires boundary conditions or intermediate outputs from other simulations to complete its calculations. As a result, efficient data exchange mechanisms are essential to maintain the fidelity and accuracy of interdependent simulations. Traditionally, data transfer between coupled models has been achieved through the use of temporary files stored on disk. However, this file-based approach imposes considerable overhead due to input/output operations and data duplication. To address these challenges, contemporary frameworks employ advanced communication libraries such as the Message Passing Interface (MPI) [ 2 ], which facilitate direct data transfer between programs. As a widely adopted standard in parallel computing, MPI offers robust communication functions that enable efficient data exchange in high-performance computing environments, thereby significantly reducing the overhead associated with file-based methods and accelerating simulation workflows. The Code-To-Code Adapter (CoToCoA) [ 3 ] library has been developed as a specialized framework that facilitates direct communication between different STP simulation codes, eliminating the necessity for intermediate file exchanges. This capability is particularly advantageous for coupled computations, where seamless interoperability among disparate software packages is required to model complex STP phenomena. At present, CoToCoA supports Fortran and C environments and utilizes the MPI library to manage inter-program communication [ 4 ]. The integration of Python with CoToCoA is motivated by the prominent role that Python plays in scientific computing, data analysis, and the development of machine learning and artificial intelligence (AI) applications. The current limitations of CoToCoA to C and Fortran environments restrict its applicability in emerging research areas that demand advanced data analytics and machine learning techniques. Thus, the objective of this study is to extend CoToCoA through the development of a Python interface, thereby overcoming these constraints. The introduction of Python support enables CoToCoA to interact with a broad spectrum of Python libraries, enhancing its versatility and facilitating seamless data exchange across programming languages. This integration combines the computational efficiency of C and Fortran with the powerful data processing and analysis capabilities offered by Python. In addition to CoToCoA, several other frameworks have been established to address the challenges inherent in cross-reference and multi-physics simulations. For example, preCICE is a fully parallel library designed for the coupling of single-physics solvers in multi-physics simulations [ 5 ]. It supports bidirectional communication between solvers and employs both explicit and implicit coupling strategies to ensure numerical stability and convergence. Advanced data mapping techniques in preCICE facilitate efficient integration of solvers with non-matching grids, while its decentralized communication model enhances scalability for large-scale parallel computations. Whereas CoToCoA prioritizes efficient data exchange between parallel codes with minimal overhead, preCICE emphasizes sophisticated numerical and communication strategies for robust multi-physics coupling. The Multiphysics Object-Oriented Simulation Environment (MOOSE) [ 6 ] is an open-source, high-performance C + + finite element toolkit developed by Idaho National Laboratory. As described in "Physics-based multiscale coupling for full core nuclear reactor simulation", MOOSE streamlines the development of fully coupled, nonlinear multiphysics applications. Its modular architecture, featuring the MultiApp and Transfer systems, supports parallel simulations and efficient data exchange across different physical domains and scales. These features facilitate the integration of various domain-specific solvers, such as those for neutron transport, thermal-fluid dynamics, and structural mechanics, into unified simulation frameworks. In comparison, CoToCoA is optimized for efficient, low-overhead data exchange, focusing primarily on cross-language and cross-code interactions, while MOOSE excels in constructing complex, monolithic simulation environments. Despite their advanced capabilities, these frameworks generally offer limited support for Python integration. Although preCICE provides partial Python compatibility, its primary emphasis on surface coupling constrains its utility for a broader range of cross-reference simulations. Both MOOSE and CoToCoA lack native Python interfaces, which can limit their effectiveness for tasks involving data analysis, machine learning, and AI. This underscores the necessity of Python integration within modern cross-reference simulation frameworks. Furthermore, the use of the ctypes library in this context ensures that the developed Python interface achieves performance levels comparable to those of C, making it suitable for computationally demanding applications. Another widely adopted solution is SWIG (Simplified Wrapper and Interface Generator) [ 7 ], which streamlines the process of connecting C or C + + code to high-level languages such as Python. SWIG automatically generates wrapper code, allowing developers to expose functions, classes, and variables from C or C + + libraries to Python with minimal manual intervention. This capability is particularly valuable in large-scale projects that require integrating extensive external libraries into Python-based workflows. In this study, two methodologies were employed to develop the Python interface, and their data transfer performance was systematically evaluated. The first approach involved implementing CoToCoA functionality directly in Python utilizing the MPI “mpi4py” library. The second approach utilized Python’s “ctypes” library to enable Python programs to access CoToCoA functions implemented in C. Following the development phase, comprehensive performance evaluations were conducted on both types of data transfer functions within CoToCoA. To demonstrate the practical utility of the developed Python interface, it was applied to enable real-time visualization of simulation results generated by a magnetohydrodynamics (MHD) simulation code for planetary magnetospheres [ 8 ]. 2 Overview of the CoToCoA CoToCoA is a specialized framework designed to facilitate coupled computations, such as those required in multi-scale simulations, by enabling direct communication among multiple programs without the use of intermediate files [ 4 ]. The overall architecture of CoToCoA comprises three distinct types of programs: the “Requester,” the “Coupler,” and the “Worker,” as illustrated in Fig. 1 . Both the Requester and Worker are implemented as multi-process programs, while the Coupler operates as a single-process moderator between the two. The Requester and Worker each manage separate simulation models and function within independent, non-overlapping process groups, both executing as standard MPI programs. The Coupler coordinates the interactions and communication between the Requester and Worker [ 9 ]. The original version of CoToCoA possesses the following core features [ 4 ]: Unidirectional data flow. Minimal modifications required for integration with existing codes. Support for asynchronous execution. One-sided data transfer capability. Developed in the C programming language. Compatibility with both C and Fortran codes. Within CoToCoA, the data flow is strictly unidirectional, proceeding from the Requester to the Worker, analogous to scenarios such as the transmission of solar wind parameters to a magnetosphere model. The framework permits a single Requester to communicate with multiple Workers, which may represent, for example, multiple spacecraft within the magnetosphere. The Requester has the ability to select a specific Worker and transmit the necessary requests and data to that Worker for processing. In contrast to conventional methods that employ temporary file transfers, CoToCoA introduces a Coupler Program to substantially reduce the modifications required for integrating existing simulation codes. This strategy enables direct memory-to-memory data transfer, eliminating the input/output overhead associated with file-based exchanges [ 4 ]. During data transfer from the Requester to the Worker, the Coupler acts as an intermediary, forwarding and temporarily storing requests (operations) and associated data. Upon receiving a request from the Requester, the Coupler dispatches it to an available Worker. If no Worker is available, the Coupler temporarily places the request and data into a queue until processing resources become available. Process synchronization within CoToCoA is maintained through three principal states: “Fin,” “Idle,” and “Running.” Each process verifies its current state prior to executing any operation, raising an error if the state is not appropriate. Furthermore, each Worker alternates between “Busy” and “Idle” states, which the Coupler uses to determine the allocation of incoming requests. As depicted in Fig. 2 , all processes and Workers are initialized in the Idle state. The Requester initiates the computation, while the Coupler non-blockingly monitors for messages from the Requester using the MPI_Iprobe function, which checks the computational status. Workers remain in a waiting state for computation requests from the Coupler. Upon receipt of a request, a Worker transitions to the Busy state and begins its computation. Once the computation is completed, the Worker returns to the Idle state. The Coupler continuously monitors the states of the Workers and manages the routing of requests and data from the Requester. When a request is received, the Coupler identifies the appropriate Worker based on its identifier and forwards the request accordingly. If all Workers are occupied, the Coupler temporarily stores the request in a queue. The Coupler periodically checks this queue and, when a Worker becomes available, retrieves and dispatches the request. After the Requester completes its computations and the transmission of requests, it sets its state to “Fin” and communicates this status to the Coupler. When the Coupler detects that all Requester processes have reached the terminated state and the request queue is empty, it transitions its own state to “Fin” and propagates this status to the Workers. Upon completing their assigned tasks, the Workers also transition to the “Fin” state. CoToCoA additionally supports one-sided communication, wherein the Requester can make a specified region of its memory accessible for direct read or write operations by the Coupler or Worker [ 4 ]. This mechanism further enhances the efficiency of inter-program data exchange by circumventing the need for intermediary storage. 3 Development of the Python Interface The development of the Python interface for CoToCoA was undertaken using two distinct methodologies. The first approach involved direct implementation in Python utilizing the "mpi4py" library, while the second leveraged the "ctypes" module to access original CoToCoA functions from Python. With these interfaces, Python programs can function as Workers within the CoToCoA framework. This enhancement broadens the applicability of CoToCoA, enabling integration with Python’s extensive ecosystem for visualization, data analysis, and machine learning. The mpi4py package provides Python bindings for the MPI standard, enabling parallel computations within Python applications and facilitating direct communication between CoToCoA and Python. This approach utilizes Python’s native capabilities for managing data structures and parallel processes, allowing seamless integration with the MPI-based communication mechanisms inherent to CoToCoA. The ctypes library, an external function interface for Python, offers C-compatible data types and facilitates the calling of functions from dynamic link libraries (DLLs) or shared libraries. This method enables the encapsulation of CoToCoA’s C functions within Python, allowing Python programs to access CoToCoA functionality without extensive alterations to the original C code. 3.1 Overview of Python Interface For the implementation of the Python interface, a dedicated Python class named "CTCAWorker" was designed. This class enables Python applications to participate as Workers within the CoToCoA framework. Both the direct development using mpi4py and the integration via the ctypes library were employed in the construction of CTCAWorker. 3.1.1 Direct Development with mpi4py In the mpi4py-based implementation, both Requester and Worker programs utilize MPI, as illustrated in Fig. 3 . The Worker operates on the Python side using mpi4py, which manages communication functions. Internal array operations are handled by the NumPy module, enabling efficient manipulation of arrays compatible with both C and Fortran types. This extension is particularly advantageous for scientific computing, ensuring robust communication and data processing capabilities. 3.1.2 Development Using ctypes The integration of Python with C-based libraries in CoToCoA using ctypes consists of the following steps: Compile the Worker C code as a shared library: Convert the C code responsible for Worker functions into a shared library format suitable for loading within Python. Load the shared library: Employ the Python ctypes module to load the compiled shared library. Specify argument and return types: Define the argument and return types of the C functions prior to their use in Python to ensure accurate data handling between the two languages. Call the C function: After configuring types and loading the library, C functions can be used within Python in the same manner as native Python methods. Utilizing ctypes for interface development does not necessitate modification of the original CoToCoA code. When employing the CTCAWorker Python interface, only the relevant parameters must be provided to implement Worker functionality, as illustrated in Fig. 4 . This methodology facilitates straightforward extension to directly use the C-based Worker. 3.2 CTCAWorker Class The CTCAWorker class encompasses the following methods (corresponding to functions in C or Fortran): Initialization: Set up the Worker’s MPI environment and generate commonly used tables. Poll request: Retrieve computation requests from the Coupler. Process control: Manage and coordinate the workflow within CoToCoA. Registration of the area: Enable access to designated regions of the Requester’s memory. Read/write data: Facilitate reading and writing operations from the registered area of the Requester. Profile: Collect execution time metrics. 3.2.1 Initialization The initialization method is responsible for preparing the Worker program for operation within CoToCoA. It accepts two arguments: the Worker program’s identifier, which ensures unique identification, and the number of processes assigned to the Worker. Upon execution, the Worker’s MPI environment is established. The initialization routines for the Requester, Coupler, and Worker are executed sequentially to establish the inter-program communication environment. 3.2.2 Poll Request This method enables the transfer of data along with computation requests. It is suitable for scenarios where the Requester has already generated the data to be transmitted. The Coupler is essential for facilitating this data transfer from the Requester to the Worker. The "pollreq" function is designed to retrieve requests from the Coupler within the CoToCoA framework. It utilizes "MPI_Recv" to receive data and requests originating from the Requester. The data transfer process commences with the Requester transmitting data to the Coupler, which then stores the data in a queue. The Coupler iterates through this queue, identifies available Worker processes, and sends the data accordingly. Subsequently, Worker processes use the pollreq function to obtain the data from the Coupler. 3.2.3 Read Data from the Requester Registered Area CoToCoA also facilitates one-sided communication, permitting direct access to the registered memory area (array) of the Requester program by the Coupler or Worker. The "readarea" function is developed to enable direct reading from the Requester’s registered memory area. It utilizes "MPI_Get," which is a component of MPI’s one-sided communication protocol. This approach allows Coupler or Worker processes to access memory directly without synchronization between the Coupler/Worker and the Requester. While this communication method is highly efficient, it is important to ensure that the registered area has been updated prior to performing read operations. 3.2.4 Profile The CoToCoA framework incorporates performance evaluation routines to measure the execution times of various components. These functions utilize "MPI_Wtime" to obtain CPU time measurements. As depicted in Fig. 5 , the framework records three principal timings: the total duration of communication initiated by the Requester, the computation time within the Worker, and the execution time for each function within CoToCoA. Upon completion, the measured times are transmitted to the Requester, and the results are output at the conclusion of the Requester program. 4 Performance Evaluation The Python interface was developed by defining the CTCAWorker class as outlined in Section 3 , with two implementation strategies. This section presents a detailed comparison and assessment of these two Python implementations and the original C version, focusing on computational efficiency and suitability for high-performance applications. 4.1 Measurement Setting Performance evaluations were conducted on a single node of System B: Laurel 3, a supercomputer system at Kyoto University [ 10 ]. The specifications of Laurel 3 are summarized in Table 1. For the Python interface, Python 3.8 and mpi4py were employed. The original CoToCoA was compiled using "mpiicc (intel/2023.2)", which is specifically designed for the Intel MPI environment. The evaluation encompassed three interface types: the original C language interface of CoToCoA, the Python interface using mpi4py, and the Python interface using ctypes. The Python interfaces introduce additional overhead relative to the C interface, particularly during data transfer operations. To quantify this overhead and assess communication efficiency, performance measurements were conducted for two data transfer functions, pollreq and readarea. These measurements facilitate a comprehensive comparison of each interface’s ability to manage data communication tasks. 4.2 pollreq function To evaluate the performance of the pollreq function in CoToCoA, the Requester and Coupler programs were implemented in C, while the Worker programs utilized three distinct interfaces to receive data: the C language interface, the Python interface with mpi4py, and the Python interface with ctypes. Figure 6 illustrates the execution times for the pollreq function across varying data sizes. The horizontal axis represents data sizes from 1 MB to 16 MB, and the vertical axis indicates execution time in milliseconds. Each test was repeated ten times to minimize experimental error, and the mean value was used for analysis. For smaller data sizes (1 MB and 2 MB), the Python interface using ctypes demonstrated the shortest execution times (2.058 ms and 4.955 ms), outperforming both the C interface (2.859 ms and 5.416 ms) and the Python interface using mpi4py (2.473 ms and 5.127 ms). At medium data sizes (4 MB and 8 MB), execution times increased for all implementations, and the C interface began to show improved performance. At 4 MB, the ctypes implementation remained fastest (6.825 ms), followed by C (7.304 ms) and mpi4py (7.397 ms). At 8 MB, execution times for the C (11.630 ms) and ctypes (11.633 ms) implementations were nearly identical, while mpi4py required more time (12.375 ms). At the largest tested data size (16 MB), the C implementation achieved the shortest execution time (18.430 ms). The ctypes implementation (19.128 ms) exhibited performance close to C, while the mpi4py implementation required more time (19.510 ms). 4.3 readarea function The performance of the readarea function was assessed using C for the Requester and Coupler programs, as in the evaluation of pollreq. Worker programs employed the three interfaces to access data from the Requester's memory. Figure 7 presents the execution times for the readarea function across data sizes from 1 MB to 16 MB, with the mean values calculated from ten measurements. The C implementation consistently produced the lowest execution times for all data sizes, reflecting its native integration with the MPI environment and minimal overhead. The mpi4py implementation required significantly more time than the C implementation, highlighting the additional complexity associated with Python and mpi4py. The ctypes implementation performed competitively, especially as data sizes increased. For smaller data sizes (1 MB), ctypes (0.763 ms) outperformed mpi4py (0.949 ms). As data sizes increased, the ctypes implementation showed better scalability and improved efficiency relative to mpi4py. At 16 MB, ctypes (4.376 ms) was faster than mpi4py (4.920 ms) and closely matched C (4.215 ms), indicating that ctypes offers lower runtime overhead and performance approaching that of the C implementation, particularly for larger data volumes. Among the Python implementations, the mpi4py interface required more execution time than C, especially with increasing data sizes. The ctypes implementation generally demonstrated improved performance compared to mpi4py, particularly for larger data sizes. 4.3 Performance Comparison between pollreq and readarea Figure 8 presents a comparative analysis of execution times for pollreq and readarea across the three interfaces: the C language interface, the Python interface with mpi4py, and the Python interface with ctypes. The data are derived from the results shown in Figs. 6 and 7 . The execution times for pollreq were substantially greater than those for readarea, indicating that pollreq involves more processing overhead. When comparing the two functions, pollreq required more time than readarea for all interfaces and data sizes. Furthermore, the gap in performance between pollreq and readarea widened as data sizes increased. This can be attributed to the additional complexity of pollreq, which includes probing, request management, and communication-related overhead. In contrast, readarea is centered on direct memory access and data transfer, which eliminates the necessity to wait for a request to be sent. 5 Discussion Analysis of the pollreq function reveals that, for data sizes of 1 MB and 2 MB, both Python-based interfaces achieve shorter execution times than the C interface. However, as the volume of data increases, execution times for both Python interfaces exceed those observed for the C implementation. Notably, within the 1 MB to 4 MB data range, the Python interface utilizing ctypes exhibits the shortest execution times. This outcome is somewhat unexpected, considering that the ctypes interface typically introduces overhead due to the necessity of calling C functions from Python. Therefore, it was not anticipated that the ctypes-based Python interface would demonstrate superior execution speed compared to the native C implementation. This unexpected observation may be explained by the operational workflow within the CoToCoA framework’s Worker process. As depicted in Fig. 9 , when the pollreq function is executed by the Worker, it internally calls the MPI_Probe function to wait for incoming data from the Coupler. Accordingly, the measured execution time of the function encompasses both the waiting period and the data transmission phase. Furthermore, prior to execution of pollreq, the C program completes initialization and preparatory operations more efficiently, thereby entering the waiting state earlier. This sequence of events may lead to inflated measured runtimes for the C implementation. As data size increases, the performance of the Python interface based on ctypes converges with that of the C interface. These findings indicate that ctypes provides an effective bridge between the high-level accessibility of Python and the efficiency of C, particularly for larger data volumes. The analysis presented above demonstrates that Python interfaces do not match the execution performance of the native C interface. Nevertheless, due to the architectural design of the CoToCoA framework, the observed differences in data transfer speed when utilizing the Python interface do not significantly affect the overall system operation. This robustness is primarily attributed to the Coupler’s use of MPI_Iprobe and a request queue, which together enable non-blocking reception of requests from the Requester. Consequently, the Requester is not required to wait for a Worker to complete its operation before proceeding with subsequent requests. With respect to the readarea function, the C interface consistently achieves the shortest execution times for all tested data sizes, underscoring its high efficiency for data transfer operations. The Python interface based on mpi4py exhibits longer execution times compared to C, particularly as the data size increases. In contrast, the ctypes-based Python interface demonstrates performance that closely approaches that of the C implementation across all examined data volumes. The relatively slower performance of the Python interfaces, particularly mpi4py, can be attributed to the additional abstraction layers and overhead associated with connecting Python to MPI operations. The ctypes-based approach mitigates some of this overhead by supporting direct calls to C library functions, resulting in execution times that more closely match those of the C implementation. Nonetheless, some performance penalties persist, reflecting the unavoidable costs of crossing the language boundary. For the readarea function, although differences in performance exist among the various interfaces, the use of MPI_Get enables direct access to the Requester’s memory, resulting in reduced execution times. Consequently, while the Python interfaces introduce some overhead, this does not impact the actual numerical simulation process. The function’s efficiency in data transfer ensures that use of the Python interface does not negatively affect the overall performance of the simulation. When comparing pollreq and readarea, for each data size from 1 MB to 16 MB, the readarea function consistently exhibits shorter execution times. Specifically, the C interface demonstrates the highest performance, followed by the Python interface using ctypes, and then the Python interface using mpi4py. As the data size increases, the difference in execution time between pollreq and readarea becomes more pronounced. The pollreq function requires more execution time than readarea across all interfaces, especially as the data volume increases. This result is explained by the workflow of pollreq, which is based on MPI_Recv and obliges the Worker to wait for requests from the Requester before proceeding, thereby introducing additional latency. In contrast, readarea is built on MPI_Get, a component of MPI one-sided communication, permitting direct access to the memory of the Requester and eliminating the need to wait for a request to be sent. 6 Application to Visualization of MHD Simulations The Python interface developed for the CoToCoA framework has been designed for real-time visualization and machine learning applications, as well as the analysis of simulation outputs using artificial intelligence algorithms. To demonstrate the capability and practicality of the Python interface, it was applied to the visualization of results from a magnetohydrodynamics (MHD) simulation code [5]. Within this visualization framework, the MHD simulation code functions as the Requester, while a Python program serves as the Worker, performing post-processing and visualization of simulation data through the developed interface. The architecture for visualizing MHD simulation results using CoToCoA is illustrated in Figure 10. In this structure, the Requester, implemented in Fortran, executes the MHD simulation computations. The Coupler, also written in Fortran, aggregates data from each process associated with the Requester and constructs the complete grid dataset. Subsequently, the Worker, implemented as a Python program, receives the consolidated grid data from the Coupler and performs visualization tasks. In this workflow, each process of the MHD simulation initiates a calculation loop within the Requester. Upon completing each computational step, the process transmits the global index corresponding to its computational domain to the Coupler. Simultaneously, the Coupler monitors for incoming data from the Requester through the MPI_Iprobe function. Upon receipt of index information, the Coupler retrieves the relevant data from the memory of the Requester process and incorporates this data into the global grid using the provided indices. Once all process data has been collected, the complete grid is assembled and placed into a queue. The Coupler then transmits the queued data to the Python-based Worker for visualization. This structure ensures that the calculation of each step in the MHD simulation proceeds independently, without requiring the Worker to complete its visualization task before the next step commences. Each process is responsible solely for transmitting index information, thereby optimizing communication time. The core structure of the visualization framework is presented in Figure 11. 6.1 Results of the Visualization Utilizing this visualization framework, the MHD simulation program achieves real-time visualization without the need for file-based input and output operations. Figure 12 presents the visualization results. Each process within the MHD simulation program, acting as a Requester, transmits its local grid computation results to the Coupler after a predefined number of computational steps. The Coupler aggregates the local grid data from the memory of the Requester processes, composes the global grid, and forwards this information to the Python program. The Python program subsequently generates visualizations of the global grid using the Matplotlib library. The MHD simulation used here has a grid size of . If written to disk, a single output would be about 6 GB. In the test, 268,800 time steps were performed, corresponding to 210 minutes of simulated time. The MHD simulation (Requester) was coupled with Python-based visualization (Worker) using CoToCoA, and visualization was performed at every time step. The data are copied directly from the Requester’s memory to the Worker’s memory, so the simulation produces no file output. With file-based visualization, the required storage would be time steps (about 1,600 TB). Since current production runs are more than two orders of magnitude larger, this approach is not practical at present. CoToCoA makes it possible to visualize such runs without generating time-step files. For this problem size, one output takes about 0.08 s with distributed I/O using 448 processes (about 14 MB per process). Writing all time steps would therefore take (about 6 hours). This I/O time is avoided by using CoToCoA for in-situ visualization. Beyond visualization, the described framework is structured to support direct integration with machine learning and artificial intelligence models during simulation. By substituting the visualization component of the Python Worker with an AI learning module, simulation data can be utilized for real-time training of AI models. For instance, grid data produced at each computational timestep can be harnessed to train AI models for predictive analysis or optimization. This configuration enhances the efficiency and speed of combining simulations with AI-driven learning processes. The integration obviates the requirement for intermediate file storage, thereby supporting high-frequency data analysis at every computational step. 7 Conclusion This research presents the design and comprehensive assessment of a Python interface for the CoToCoA framework, advancing its applicability within contemporary computational science. Recognizing the prominent role of Python in scientific computing, data analysis, machine learning, and artificial intelligence, the development of a Python interface for CoToCoA strengthens interoperability between programming languages and merges the computational efficiency of C with Python’s robust data processing capabilities. Two principal approaches were examined in developing the Python interface: direct implementation using mpi4py and the utilization of ctypes for accessing C functions within CoToCoA from Python. This interface enables Python programs to operate as Workers, processing data transmitted from Requesters. Performance evaluations were conducted by testing the data transfer functions of CoToCoA, specifically pollreq and readarea. Results demonstrate that, for pollreq, the Python interface employing ctypes approaches the performance of the C interface as data size grows. Conversely, for readarea, the Python interface with mpi4py exhibits greater execution times compared to C, particularly with increasing data volume. The ctypes-based Python interface consistently approximates the C interface’s performance across varying data sizes. The comparative analysis of pollreq and readarea reveals that readarea consistently achieves lower execution times. This outcome is attributed to its foundation on MPI_Get, an element of MPI one-sided communication that permits direct memory access from the Requester. In contrast, pollreq is structured around MPI_Recv, which requires the Worker to await incoming requests before proceeding, introducing additional latency. The evaluation indicates that the Python interface utilizing ctypes achieves performance comparable to C, as it enables direct function calls to C libraries. This finding demonstrates that ctypes effectively bridges the gap between Python’s high-level accessibility and the low-level efficiency of C, especially with large datasets. The developed Python interface was further applied to the visualization of simulation results from a magnetohydrodynamics (MHD) simulation code, thereby demonstrating its practical utility. Through this application, the framework has shown significant potential for integration with Python-based tools, including machine learning and artificial intelligence models. By leveraging Python’s extensive data processing and visualization libraries, the interface enables the use of advanced analytical tools in simulation analysis. Furthermore, the scope of this interface is anticipated to expand to applications in deep learning and artificial intelligence. Building upon the current visualization capabilities, future research may pursue direct transmission of simulation data to machine learning models during runtime. This approach would enable real-time interaction between simulations and ML/AI models, supporting tasks such as model training, anomaly detection, and prediction without dependence on intermediate file storage. Eliminating extensive file-based input/output operations would conserve disk space and mitigate data management overhead, thereby enhancing the speed and efficiency of integrating simulations with ML/AI methodologies. Declarations Ethics approval and consent to participate “Not applicable” in this section. Consent for publication “Not applicable” in this section. Availability of materials The CoToCoA framework including the Python interface is available in the web site (https://github.com/tnanri/cotocoa) and doi:10.5281/zenodo.5655840. Competing interests The authors declare no competing interests. Funding This work was supported by JSPS KAKENHI Grant Number 22K12049 and 25K15146. Authors' contributions K.F. designed the scientific content and Python interface, H.Z. implemented the Python interface. K. F. and H. Z. wrote this manuscript. T.N. contributed to theoretical support of CoToCoA and evaluation of Python interface. Y.M. and Y. K. contributed to the application of Python to scientific simulation. All authors discussed the results and contributed to the final manuscript. Acknowledgements This research used computational resources by Academic Center for Computing and Media Studies, Kyoto University. This study uses computational resources of the KDK computer system at the Research Institute for Sustainable Humanosphere, Kyoto University, and the HPCI system provided by the Research Institute for Information Technology, Kyushu University, and the Cyberscience Center, Tohoku University, through the HPCI System Research Project (Project IDs: hp230046, hp240018, and hp250070). This work is partly supported by the Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures (JHPCN) in Japan (Project ID: jh250014). Authors' information Research Institute for Humanity and Nature, Kyoto, Japan Keiichiro FUKAZAWA Graduate School of Informatics, Kyoto University, Kyoto, Japan Haichao ZHAO Research Institute for Information Technology, Kyushu University, Fukuoka, Japan Takeshi NANRI Graduate School of System Informatics, Kobe University, Kobe, Japan Yohei MIYAKE Graduate School of Science Department of Geophysics, Tohoku University, Sendai, Japan Yuto Katoh References John T. Hwang and Joaquim R.R.A. Martins. 2018. A Computational Architecture for Coupling Heterogeneous Numerical Models and Computing Coupled Derivatives. ACM Trans. Math. Softw. 44, 4, Article 37 (December 2018), 39 pages. https://doi.org/10.1145/3182393. W. Gropp, E. Lusk, and A. Skjellum. Using MPI: portable parallel programming with the message-passing interface, vol. 1. MIT press, 1999. K. Fukazawa, Y. Katoh, T. Nanri, and Y. Miyake. Application of cross-reference framework cotocoa to macro- and micro-scale simulations of planetary magnetospheres. In 2019 Seventh International Symposium on Computing and Networking Workshops (CANDARW), pp. 121--124, 2019. T. Nanri. Code-to-code adapter. https://github.com/tnanri/cotocoa,2024. Accessed: 2024-12-23. H. Bungartz, F. Lindner, B. Gatzhammer, M. Mehl, K. Scheufele, A. Shukaev, and B. Uekermann. precice-a fully parallel library for multi-physics surface coupling. Computers \& Fluids, vol. 141, pp. 250-258, 2016. D. R Gaston, C. J Permann, J. W Peterson, A. E Slaughter, D. Andres, Y. Wang, M. P Short, D. M Perez, M. R Tonks, and J. Ortensi. Physics-based multiscale coupling for full core nuclear reactor simulation. Annals of Nuclear Energy, vol. 84, pp. 45--54, 2015. D. M Beazley. Swig: An easy to use tool for integrating scripting languages with c and c++. In Tcl/Tk Workshop, vol. 43, pp. 74, 1996. K. Fukazawa, T. Ogino, and R. J Walker. Configuration and dynamics of the Jovian magnetosphere. Journal of Geophysical Research: Space Physics, 111(A10), 2006. Y. Miyake, Y. Sunada, Y. Tanaka, K. Nakazawa, T. Nanri, K. Fukazawa, and Y. Katoh. Implementation of coupled numerical analysis of magnetospheric dynamics and spacecraft charging phenomena via code-to-code adapter (cotocoa) framework. In International Conference on Computational Science, pp. 438--452. Springer, 2023. System Specification of Supercomputer at Kyoto University (2023) (https://www.iimc.kyoto-u.ac.jp/en/services/comp/supercomputer/system/specification) Table Table 1 is available in the Supplementary Files section. Supplementary Files table1.pdf Table 1. Node specifications of System B: Laurel. abstract.jpg Cite Share Download PDF Status: Under Review Version 1 posted Reviewers agreed at journal 20 Apr, 2026 Reviewers invited by journal 20 Apr, 2026 Editor assigned by journal 10 Apr, 2026 First submitted to journal 07 Apr, 2026 Editorial decision: Major Revision 02 Apr, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9195901","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":625971099,"identity":"38433ad6-9357-466c-ae20-3fd20c6fe2b1","order_by":0,"name":"Keiichiro FUKAZAWA","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA8ElEQVRIiWNgGAWjYBAC9gY2CIOfgYfhYANU1ACfFp4DUC2SDSRrMTjAw8DYgE8pXItEWuLHHzWH5Y2Pnz14cGbbYQb+9gMMxQX4tRyWkDh22HDbmbyEgxuBWiTOJDAYz8CjxV4ivUHCgO0247YbPAYHHwK1MNxgYDDmwWtLevOPhH+37TfPgGqRJ6wl7ZjEwbbbiRskgFpADjMgqIXnWZplY9//5BlncgwOzjiXzmN4JrEBr1942NOMb/74lmbb337G+GNPmbWc3PHDx4zxhRgqYGQDpgEGxjZjonUwMPwBk8yPSdAyCkbBKBgFwx8AAMHDUgtKrNBIAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-4862-6411","institution":"Research Institute for Humanity and Nature: Sogo Chikyu Kankyogaku Kenkyujo","correspondingAuthor":true,"prefix":"","firstName":"Keiichiro","middleName":"","lastName":"FUKAZAWA","suffix":""},{"id":625971100,"identity":"520109fc-106c-43dc-957a-5bb33c807586","order_by":1,"name":"Haichao ZHAO","email":"","orcid":"","institution":"Kyoto University: Kyoto Daigaku","correspondingAuthor":false,"prefix":"","firstName":"Haichao","middleName":"","lastName":"ZHAO","suffix":""},{"id":625971101,"identity":"c55118ce-7b0b-459d-a390-63feeaaab30a","order_by":2,"name":"Takeshi NANRI","email":"","orcid":"","institution":"Kyushu University: Kyushu Daigaku","correspondingAuthor":false,"prefix":"","firstName":"Takeshi","middleName":"","lastName":"NANRI","suffix":""},{"id":625971102,"identity":"4b85ce1a-a105-42a0-a39f-764be8bd53f3","order_by":3,"name":"Yohei MIYAKE","email":"","orcid":"","institution":"Kobe University: Kobe Daigaku","correspondingAuthor":false,"prefix":"","firstName":"Yohei","middleName":"","lastName":"MIYAKE","suffix":""},{"id":625971103,"identity":"311fffbf-09b1-486e-be77-8553bcf20bd4","order_by":4,"name":"Yuto KATOH","email":"","orcid":"","institution":"Tohoku University: Tohoku Daigaku","correspondingAuthor":false,"prefix":"","firstName":"Yuto","middleName":"","lastName":"KATOH","suffix":""}],"badges":[],"createdAt":"2026-03-23 05:41:03","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9195901/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9195901/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108017869,"identity":"ca2d8716-c70c-42cd-9a3a-c0d6a03d6703","added_by":"auto","created_at":"2026-04-28 13:56:24","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":277823,"visible":true,"origin":"","legend":"\u003cp\u003eOverview of code coupling via CoToCoA\u003c/p\u003e","description":"","filename":"fig1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/2c2a2a81074b69e5525a6687.jpg"},{"id":108017886,"identity":"9ace3d08-f5b3-403c-bf74-4cbbf24ac369","added_by":"auto","created_at":"2026-04-28 13:56:28","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":389520,"visible":true,"origin":"","legend":"\u003cp\u003eTypical coordinative behavior of Requester, Coupler, and Workers within the CoToCoA framework\u003c/p\u003e","description":"","filename":"fig2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/51a540a6d087618248b6d356.jpg"},{"id":108017890,"identity":"a8da30ab-5418-43a1-8eeb-e7730758479e","added_by":"auto","created_at":"2026-04-28 13:56:29","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":218834,"visible":true,"origin":"","legend":"\u003cp\u003ePython interface developed with mpi4py\u003c/p\u003e","description":"","filename":"fig3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/a1b46608ddb98c43f05bcae4.jpg"},{"id":108017870,"identity":"568d49ea-5287-430c-adc6-7a01fc183759","added_by":"auto","created_at":"2026-04-28 13:56:24","extension":"jpg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":197800,"visible":true,"origin":"","legend":"\u003cp\u003ePython interface developed with ctypes\u003c/p\u003e","description":"","filename":"fig4.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/7cace8da6caf6d68c4444f4b.jpg"},{"id":108017873,"identity":"64e88586-0d7d-4ea1-8be1-b0299d743537","added_by":"auto","created_at":"2026-04-28 13:56:24","extension":"jpg","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":330052,"visible":true,"origin":"","legend":"\u003cp\u003eTiming Analysis in CoToCoA Framework\u003c/p\u003e","description":"","filename":"fig5.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/6d8ebd8b6590035195cc7c6e.jpg"},{"id":108017885,"identity":"072b6bf1-d485-432d-ab2a-9732b32a5d41","added_by":"auto","created_at":"2026-04-28 13:56:28","extension":"jpg","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":292851,"visible":true,"origin":"","legend":"\u003cp\u003eThe performance evaluation of the pollreq function across three different interfaces\u003c/p\u003e","description":"","filename":"fig6.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/b02f982b7f7db7f2bba51def.jpg"},{"id":108017872,"identity":"2a31bc53-bd09-455e-8e7c-e6d8b3fa794e","added_by":"auto","created_at":"2026-04-28 13:56:24","extension":"jpg","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":308545,"visible":true,"origin":"","legend":"\u003cp\u003eThe performance evaluation of the readarea function across three different interfaces\u003c/p\u003e","description":"","filename":"fig7.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/f63e673bb16eb3993b90ca53.jpg"},{"id":108017874,"identity":"c949d983-7e96-4350-af9d-5932cb6687c8","added_by":"auto","created_at":"2026-04-28 13:56:24","extension":"jpg","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":374655,"visible":true,"origin":"","legend":"\u003cp\u003eCompare the performance of the pollreq and readarea functions across different interfaces\u003c/p\u003e","description":"","filename":"fig8.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/b8cb5ab61c7e4991a4338e82.jpg"},{"id":108017895,"identity":"35dc7a2a-9162-4ff4-bdac-09eb03d55211","added_by":"auto","created_at":"2026-04-28 13:56:32","extension":"jpg","order_by":9,"title":"Figure 9","display":"","copyAsset":false,"role":"figure","size":274459,"visible":true,"origin":"","legend":"\u003cp\u003eAnalysis of the process of pollreq function\u003c/p\u003e","description":"","filename":"fig9.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/ff746764a4dadfe346dac53f.jpg"},{"id":108017871,"identity":"8e7a6f6a-fe3e-4097-816e-ee40d3faf51e","added_by":"auto","created_at":"2026-04-28 13:56:24","extension":"jpg","order_by":10,"title":"Figure 10","display":"","copyAsset":false,"role":"figure","size":458390,"visible":true,"origin":"","legend":"\u003cp\u003eStructure for visualization\u003c/p\u003e","description":"","filename":"fig10.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/b71ec4ede12417ca2960c706.jpg"},{"id":108017887,"identity":"2a5a7fbf-ac33-44e5-9875-4dc0abaf89ee","added_by":"auto","created_at":"2026-04-28 13:56:28","extension":"jpg","order_by":11,"title":"Figure 11","display":"","copyAsset":false,"role":"figure","size":104694,"visible":true,"origin":"","legend":"\u003cp\u003eSkeleton code of the visualization framwork\u003c/p\u003e","description":"","filename":"fig11.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/944d464139d17715b13e726e.jpg"},{"id":108181113,"identity":"a2b2a53f-7a40-4a7b-9e7f-480954950fe0","added_by":"auto","created_at":"2026-04-30 08:57:33","extension":"jpg","order_by":12,"title":"Figure 12","display":"","copyAsset":false,"role":"figure","size":417830,"visible":true,"origin":"","legend":"\u003cp\u003eResult of the visualization of MHD simulation (Requester) with CoToCoA Python interface (Worker)\u003c/p\u003e","description":"","filename":"fig12.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/d5d9a49406b1cdeed5a52c1e.jpg"},{"id":109249129,"identity":"2681dc36-cd42-488e-8c15-faa2ab811d3e","added_by":"auto","created_at":"2026-05-14 08:42:47","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3851297,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/0283dd4d-b9f9-4461-beb5-7da472bb51cd.pdf"},{"id":108017876,"identity":"cbba4ab5-480b-4fcf-816f-e7d614b4810f","added_by":"auto","created_at":"2026-04-28 13:56:25","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":84962,"visible":true,"origin":"","legend":"\u003cp\u003eTable 1. Node specifications of System B: Laurel.\u003c/p\u003e","description":"","filename":"table1.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/d8e5d18770dca95370770101.pdf"},{"id":108017888,"identity":"0ec35917-9649-488a-945d-d54815887025","added_by":"auto","created_at":"2026-04-28 13:56:28","extension":"jpg","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":26295,"visible":true,"origin":"","legend":"","description":"","filename":"abstract.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9195901/v1/e3621227746ea511bb4dcd10.jpg"}],"financialInterests":"","formattedTitle":"Development of a Python Interface for a Framework of Cross-reference Solar-Terrestrial Physics Simulation","fulltext":[{"header":"1 Introduction","content":"\u003cp\u003eCoupled computations constitute a fundamental methodology in scientific research, enabling multiple simulation programs to interact collaboratively to address complex problems that exceed the capabilities of individual codes. This approach is particularly significant in the field of Solar-Terrestrial Physics (STP), where diverse physical processes operating at various scales are intrinsically linked and mutually influential.\u003c/p\u003e \u003cp\u003eWithin coupled computations, each simulation program typically addresses a distinct physical phenomenon and requires boundary conditions or intermediate outputs from other simulations to complete its calculations. As a result, efficient data exchange mechanisms are essential to maintain the fidelity and accuracy of interdependent simulations. Traditionally, data transfer between coupled models has been achieved through the use of temporary files stored on disk. However, this file-based approach imposes considerable overhead due to input/output operations and data duplication. To address these challenges, contemporary frameworks employ advanced communication libraries such as the Message Passing Interface (MPI) [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e], which facilitate direct data transfer between programs. As a widely adopted standard in parallel computing, MPI offers robust communication functions that enable efficient data exchange in high-performance computing environments, thereby significantly reducing the overhead associated with file-based methods and accelerating simulation workflows.\u003c/p\u003e \u003cp\u003eThe Code-To-Code Adapter (CoToCoA) [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e] library has been developed as a specialized framework that facilitates direct communication between different STP simulation codes, eliminating the necessity for intermediate file exchanges. This capability is particularly advantageous for coupled computations, where seamless interoperability among disparate software packages is required to model complex STP phenomena. At present, CoToCoA supports Fortran and C environments and utilizes the MPI library to manage inter-program communication [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe integration of Python with CoToCoA is motivated by the prominent role that Python plays in scientific computing, data analysis, and the development of machine learning and artificial intelligence (AI) applications. The current limitations of CoToCoA to C and Fortran environments restrict its applicability in emerging research areas that demand advanced data analytics and machine learning techniques. Thus, the objective of this study is to extend CoToCoA through the development of a Python interface, thereby overcoming these constraints. The introduction of Python support enables CoToCoA to interact with a broad spectrum of Python libraries, enhancing its versatility and facilitating seamless data exchange across programming languages. This integration combines the computational efficiency of C and Fortran with the powerful data processing and analysis capabilities offered by Python.\u003c/p\u003e \u003cp\u003eIn addition to CoToCoA, several other frameworks have been established to address the challenges inherent in cross-reference and multi-physics simulations. For example, preCICE is a fully parallel library designed for the coupling of single-physics solvers in multi-physics simulations [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. It supports bidirectional communication between solvers and employs both explicit and implicit coupling strategies to ensure numerical stability and convergence. Advanced data mapping techniques in preCICE facilitate efficient integration of solvers with non-matching grids, while its decentralized communication model enhances scalability for large-scale parallel computations. Whereas CoToCoA prioritizes efficient data exchange between parallel codes with minimal overhead, preCICE emphasizes sophisticated numerical and communication strategies for robust multi-physics coupling.\u003c/p\u003e \u003cp\u003eThe Multiphysics Object-Oriented Simulation Environment (MOOSE) [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e] is an open-source, high-performance C\u0026thinsp;+\u0026thinsp;+\u0026thinsp;finite element toolkit developed by Idaho National Laboratory. As described in \"Physics-based multiscale coupling for full core nuclear reactor simulation\", MOOSE streamlines the development of fully coupled, nonlinear multiphysics applications. Its modular architecture, featuring the MultiApp and Transfer systems, supports parallel simulations and efficient data exchange across different physical domains and scales. These features facilitate the integration of various domain-specific solvers, such as those for neutron transport, thermal-fluid dynamics, and structural mechanics, into unified simulation frameworks. In comparison, CoToCoA is optimized for efficient, low-overhead data exchange, focusing primarily on cross-language and cross-code interactions, while MOOSE excels in constructing complex, monolithic simulation environments.\u003c/p\u003e \u003cp\u003eDespite their advanced capabilities, these frameworks generally offer limited support for Python integration. Although preCICE provides partial Python compatibility, its primary emphasis on surface coupling constrains its utility for a broader range of cross-reference simulations. Both MOOSE and CoToCoA lack native Python interfaces, which can limit their effectiveness for tasks involving data analysis, machine learning, and AI. This underscores the necessity of Python integration within modern cross-reference simulation frameworks. Furthermore, the use of the ctypes library in this context ensures that the developed Python interface achieves performance levels comparable to those of C, making it suitable for computationally demanding applications.\u003c/p\u003e \u003cp\u003eAnother widely adopted solution is SWIG (Simplified Wrapper and Interface Generator) [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], which streamlines the process of connecting C or C\u0026thinsp;+\u0026thinsp;+\u0026thinsp;code to high-level languages such as Python. SWIG automatically generates wrapper code, allowing developers to expose functions, classes, and variables from C or C\u0026thinsp;+\u0026thinsp;+\u0026thinsp;libraries to Python with minimal manual intervention. This capability is particularly valuable in large-scale projects that require integrating extensive external libraries into Python-based workflows.\u003c/p\u003e \u003cp\u003eIn this study, two methodologies were employed to develop the Python interface, and their data transfer performance was systematically evaluated. The first approach involved implementing CoToCoA functionality directly in Python utilizing the MPI \u0026ldquo;mpi4py\u0026rdquo; library. The second approach utilized Python\u0026rsquo;s \u0026ldquo;ctypes\u0026rdquo; library to enable Python programs to access CoToCoA functions implemented in C. Following the development phase, comprehensive performance evaluations were conducted on both types of data transfer functions within CoToCoA.\u003c/p\u003e \u003cp\u003eTo demonstrate the practical utility of the developed Python interface, it was applied to enable real-time visualization of simulation results generated by a magnetohydrodynamics (MHD) simulation code for planetary magnetospheres [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e].\u003c/p\u003e"},{"header":"2 Overview of the CoToCoA","content":"\u003cp\u003eCoToCoA is a specialized framework designed to facilitate coupled computations, such as those required in multi-scale simulations, by enabling direct communication among multiple programs without the use of intermediate files [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. The overall architecture of CoToCoA comprises three distinct types of programs: the \u0026ldquo;Requester,\u0026rdquo; the \u0026ldquo;Coupler,\u0026rdquo; and the \u0026ldquo;Worker,\u0026rdquo; as illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. Both the Requester and Worker are implemented as multi-process programs, while the Coupler operates as a single-process moderator between the two. The Requester and Worker each manage separate simulation models and function within independent, non-overlapping process groups, both executing as standard MPI programs. The Coupler coordinates the interactions and communication between the Requester and Worker [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. The original version of CoToCoA possesses the following core features [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]:\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eUnidirectional data flow.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eMinimal modifications required for integration with existing codes.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eSupport for asynchronous execution.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eOne-sided data transfer capability.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eDeveloped in the C programming language.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eCompatibility with both C and Fortran codes.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eWithin CoToCoA, the data flow is strictly unidirectional, proceeding from the Requester to the Worker, analogous to scenarios such as the transmission of solar wind parameters to a magnetosphere model. The framework permits a single Requester to communicate with multiple Workers, which may represent, for example, multiple spacecraft within the magnetosphere. The Requester has the ability to select a specific Worker and transmit the necessary requests and data to that Worker for processing.\u003c/p\u003e \u003cp\u003eIn contrast to conventional methods that employ temporary file transfers, CoToCoA introduces a Coupler Program to substantially reduce the modifications required for integrating existing simulation codes. This strategy enables direct memory-to-memory data transfer, eliminating the input/output overhead associated with file-based exchanges [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. During data transfer from the Requester to the Worker, the Coupler acts as an intermediary, forwarding and temporarily storing requests (operations) and associated data. Upon receiving a request from the Requester, the Coupler dispatches it to an available Worker. If no Worker is available, the Coupler temporarily places the request and data into a queue until processing resources become available.\u003c/p\u003e \u003cp\u003eProcess synchronization within CoToCoA is maintained through three principal states: \u0026ldquo;Fin,\u0026rdquo; \u0026ldquo;Idle,\u0026rdquo; and \u0026ldquo;Running.\u0026rdquo; Each process verifies its current state prior to executing any operation, raising an error if the state is not appropriate. Furthermore, each Worker alternates between \u0026ldquo;Busy\u0026rdquo; and \u0026ldquo;Idle\u0026rdquo; states, which the Coupler uses to determine the allocation of incoming requests. As depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, all processes and Workers are initialized in the Idle state. The Requester initiates the computation, while the Coupler non-blockingly monitors for messages from the Requester using the MPI_Iprobe function, which checks the computational status. Workers remain in a waiting state for computation requests from the Coupler. Upon receipt of a request, a Worker transitions to the Busy state and begins its computation. Once the computation is completed, the Worker returns to the Idle state.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe Coupler continuously monitors the states of the Workers and manages the routing of requests and data from the Requester. When a request is received, the Coupler identifies the appropriate Worker based on its identifier and forwards the request accordingly. If all Workers are occupied, the Coupler temporarily stores the request in a queue. The Coupler periodically checks this queue and, when a Worker becomes available, retrieves and dispatches the request. After the Requester completes its computations and the transmission of requests, it sets its state to \u0026ldquo;Fin\u0026rdquo; and communicates this status to the Coupler. When the Coupler detects that all Requester processes have reached the terminated state and the request queue is empty, it transitions its own state to \u0026ldquo;Fin\u0026rdquo; and propagates this status to the Workers. Upon completing their assigned tasks, the Workers also transition to the \u0026ldquo;Fin\u0026rdquo; state.\u003c/p\u003e \u003cp\u003eCoToCoA additionally supports one-sided communication, wherein the Requester can make a specified region of its memory accessible for direct read or write operations by the Coupler or Worker [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. This mechanism further enhances the efficiency of inter-program data exchange by circumventing the need for intermediary storage.\u003c/p\u003e"},{"header":"3 Development of the Python Interface","content":"\u003cp\u003eThe development of the Python interface for CoToCoA was undertaken using two distinct methodologies. The first approach involved direct implementation in Python utilizing the \"mpi4py\" library, while the second leveraged the \"ctypes\" module to access original CoToCoA functions from Python. With these interfaces, Python programs can function as Workers within the CoToCoA framework. This enhancement broadens the applicability of CoToCoA, enabling integration with Python\u0026rsquo;s extensive ecosystem for visualization, data analysis, and machine learning.\u003c/p\u003e \u003cp\u003eThe mpi4py package provides Python bindings for the MPI standard, enabling parallel computations within Python applications and facilitating direct communication between CoToCoA and Python. This approach utilizes Python\u0026rsquo;s native capabilities for managing data structures and parallel processes, allowing seamless integration with the MPI-based communication mechanisms inherent to CoToCoA.\u003c/p\u003e \u003cp\u003eThe ctypes library, an external function interface for Python, offers C-compatible data types and facilitates the calling of functions from dynamic link libraries (DLLs) or shared libraries. This method enables the encapsulation of CoToCoA\u0026rsquo;s C functions within Python, allowing Python programs to access CoToCoA functionality without extensive alterations to the original C code.\u003c/p\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Overview of Python Interface\u003c/h2\u003e \u003cp\u003eFor the implementation of the Python interface, a dedicated Python class named \"CTCAWorker\" was designed. This class enables Python applications to participate as Workers within the CoToCoA framework. Both the direct development using mpi4py and the integration via the ctypes library were employed in the construction of CTCAWorker.\u003c/p\u003e \u003cdiv id=\"Sec5\" class=\"Section3\"\u003e \u003ch2\u003e3.1.1 Direct Development with mpi4py\u003c/h2\u003e \u003cp\u003eIn the mpi4py-based implementation, both Requester and Worker programs utilize MPI, as illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e. The Worker operates on the Python side using mpi4py, which manages communication functions. Internal array operations are handled by the NumPy module, enabling efficient manipulation of arrays compatible with both C and Fortran types. This extension is particularly advantageous for scientific computing, ensuring robust communication and data processing capabilities.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section3\"\u003e \u003ch2\u003e3.1.2 Development Using ctypes\u003c/h2\u003e \u003cp\u003eThe integration of Python with C-based libraries in CoToCoA using ctypes consists of the following steps:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eCompile the Worker C code as a shared library: Convert the C code responsible for Worker functions into a shared library format suitable for loading within Python.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eLoad the shared library: Employ the Python ctypes module to load the compiled shared library.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eSpecify argument and return types: Define the argument and return types of the C functions prior to their use in Python to ensure accurate data handling between the two languages.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eCall the C function: After configuring types and loading the library, C functions can be used within Python in the same manner as native Python methods.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eUtilizing ctypes for interface development does not necessitate modification of the original CoToCoA code. When employing the CTCAWorker Python interface, only the relevant parameters must be provided to implement Worker functionality, as illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e. This methodology facilitates straightforward extension to directly use the C-based Worker.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e3.2 CTCAWorker Class\u003c/h2\u003e \u003cp\u003eThe CTCAWorker class encompasses the following methods (corresponding to functions in C or Fortran):\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eInitialization: Set up the Worker\u0026rsquo;s MPI environment and generate commonly used tables.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003ePoll request: Retrieve computation requests from the Coupler.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eProcess control: Manage and coordinate the workflow within CoToCoA.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eRegistration of the area: Enable access to designated regions of the Requester\u0026rsquo;s memory.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eRead/write data: Facilitate reading and writing operations from the registered area of the Requester.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eProfile: Collect execution time metrics.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cdiv id=\"Sec8\" class=\"Section3\"\u003e \u003ch2\u003e3.2.1 Initialization\u003c/h2\u003e \u003cp\u003eThe initialization method is responsible for preparing the Worker program for operation within CoToCoA. It accepts two arguments: the Worker program\u0026rsquo;s identifier, which ensures unique identification, and the number of processes assigned to the Worker. Upon execution, the Worker\u0026rsquo;s MPI environment is established. The initialization routines for the Requester, Coupler, and Worker are executed sequentially to establish the inter-program communication environment.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section3\"\u003e \u003ch2\u003e3.2.2 Poll Request\u003c/h2\u003e \u003cp\u003eThis method enables the transfer of data along with computation requests. It is suitable for scenarios where the Requester has already generated the data to be transmitted. The Coupler is essential for facilitating this data transfer from the Requester to the Worker.\u003c/p\u003e \u003cp\u003eThe \"pollreq\" function is designed to retrieve requests from the Coupler within the CoToCoA framework. It utilizes \"MPI_Recv\" to receive data and requests originating from the Requester. The data transfer process commences with the Requester transmitting data to the Coupler, which then stores the data in a queue. The Coupler iterates through this queue, identifies available Worker processes, and sends the data accordingly. Subsequently, Worker processes use the pollreq function to obtain the data from the Coupler.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section3\"\u003e \u003ch2\u003e3.2.3 Read Data from the Requester Registered Area\u003c/h2\u003e \u003cp\u003eCoToCoA also facilitates one-sided communication, permitting direct access to the registered memory area (array) of the Requester program by the Coupler or Worker.\u003c/p\u003e \u003cp\u003eThe \"readarea\" function is developed to enable direct reading from the Requester\u0026rsquo;s registered memory area. It utilizes \"MPI_Get,\" which is a component of MPI\u0026rsquo;s one-sided communication protocol. This approach allows Coupler or Worker processes to access memory directly without synchronization between the Coupler/Worker and the Requester. While this communication method is highly efficient, it is important to ensure that the registered area has been updated prior to performing read operations.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section3\"\u003e \u003ch2\u003e3.2.4 Profile\u003c/h2\u003e \u003cp\u003eThe CoToCoA framework incorporates performance evaluation routines to measure the execution times of various components. These functions utilize \"MPI_Wtime\" to obtain CPU time measurements. As depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e, the framework records three principal timings: the total duration of communication initiated by the Requester, the computation time within the Worker, and the execution time for each function within CoToCoA. Upon completion, the measured times are transmitted to the Requester, and the results are output at the conclusion of the Requester program.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"4 Performance Evaluation","content":"\u003cp\u003eThe Python interface was developed by defining the CTCAWorker class as outlined in Section \u003cspan refid=\"Sec3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, with two implementation strategies. This section presents a detailed comparison and assessment of these two Python implementations and the original C version, focusing on computational efficiency and suitability for high-performance applications.\u003c/p\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Measurement Setting\u003c/h2\u003e \u003cp\u003ePerformance evaluations were conducted on a single node of System B: Laurel 3, a supercomputer system at Kyoto University [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. The specifications of Laurel 3 are summarized in Table\u0026nbsp;1. For the Python interface, Python 3.8 and mpi4py were employed. The original CoToCoA was compiled using \"mpiicc (intel/2023.2)\", which is specifically designed for the Intel MPI environment.\u003c/p\u003e \u003cp\u003eThe evaluation encompassed three interface types: the original C language interface of CoToCoA, the Python interface using mpi4py, and the Python interface using ctypes. The Python interfaces introduce additional overhead relative to the C interface, particularly during data transfer operations. To quantify this overhead and assess communication efficiency, performance measurements were conducted for two data transfer functions, pollreq and readarea. These measurements facilitate a comprehensive comparison of each interface\u0026rsquo;s ability to manage data communication tasks.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003e4.2 pollreq function\u003c/h2\u003e \u003cp\u003eTo evaluate the performance of the pollreq function in CoToCoA, the Requester and Coupler programs were implemented in C, while the Worker programs utilized three distinct interfaces to receive data: the C language interface, the Python interface with mpi4py, and the Python interface with ctypes. Figure\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003e illustrates the execution times for the pollreq function across varying data sizes. The horizontal axis represents data sizes from 1 MB to 16 MB, and the vertical axis indicates execution time in milliseconds. Each test was repeated ten times to minimize experimental error, and the mean value was used for analysis.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eFor smaller data sizes (1 MB and 2 MB), the Python interface using ctypes demonstrated the shortest execution times (2.058 ms and 4.955 ms), outperforming both the C interface (2.859 ms and 5.416 ms) and the Python interface using mpi4py (2.473 ms and 5.127 ms).\u003c/p\u003e \u003cp\u003eAt medium data sizes (4 MB and 8 MB), execution times increased for all implementations, and the C interface began to show improved performance. At 4 MB, the ctypes implementation remained fastest (6.825 ms), followed by C (7.304 ms) and mpi4py (7.397 ms). At 8 MB, execution times for the C (11.630 ms) and ctypes (11.633 ms) implementations were nearly identical, while mpi4py required more time (12.375 ms).\u003c/p\u003e \u003cp\u003eAt the largest tested data size (16 MB), the C implementation achieved the shortest execution time (18.430 ms). The ctypes implementation (19.128 ms) exhibited performance close to C, while the mpi4py implementation required more time (19.510 ms).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e4.3 readarea function\u003c/h2\u003e \u003cp\u003eThe performance of the readarea function was assessed using C for the Requester and Coupler programs, as in the evaluation of pollreq.\u0026nbsp;Worker programs employed the three interfaces to access data from the Requester's memory. Figure\u0026nbsp;\u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e7\u003c/span\u003e presents the execution times for the readarea function across data sizes from 1 MB to 16 MB, with the mean values calculated from ten measurements.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe C implementation consistently produced the lowest execution times for all data sizes, reflecting its native integration with the MPI environment and minimal overhead. The mpi4py implementation required significantly more time than the C implementation, highlighting the additional complexity associated with Python and mpi4py. The ctypes implementation performed competitively, especially as data sizes increased. For smaller data sizes (1 MB), ctypes (0.763 ms) outperformed mpi4py (0.949 ms). As data sizes increased, the ctypes implementation showed better scalability and improved efficiency relative to mpi4py. At 16 MB, ctypes (4.376 ms) was faster than mpi4py (4.920 ms) and closely matched C (4.215 ms), indicating that ctypes offers lower runtime overhead and performance approaching that of the C implementation, particularly for larger data volumes.\u003c/p\u003e \u003cp\u003eAmong the Python implementations, the mpi4py interface required more execution time than C, especially with increasing data sizes. The ctypes implementation generally demonstrated improved performance compared to mpi4py, particularly for larger data sizes.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003e4.3 Performance Comparison between pollreq and readarea\u003c/h2\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig8\" class=\"InternalRef\"\u003e8\u003c/span\u003e presents a comparative analysis of execution times for pollreq and readarea across the three interfaces: the C language interface, the Python interface with mpi4py, and the Python interface with ctypes. The data are derived from the results shown in Figs.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003e and \u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e7\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe execution times for pollreq were substantially greater than those for readarea, indicating that pollreq involves more processing overhead. When comparing the two functions, pollreq required more time than readarea for all interfaces and data sizes. Furthermore, the gap in performance between pollreq and readarea widened as data sizes increased. This can be attributed to the additional complexity of pollreq, which includes probing, request management, and communication-related overhead. In contrast, readarea is centered on direct memory access and data transfer, which eliminates the necessity to wait for a request to be sent.\u003c/p\u003e \u003c/div\u003e"},{"header":"5 Discussion","content":"\u003cp\u003eAnalysis of the pollreq function reveals that, for data sizes of 1 MB and 2 MB, both Python-based interfaces achieve shorter execution times than the C interface. However, as the volume of data increases, execution times for both Python interfaces exceed those observed for the C implementation. Notably, within the 1 MB to 4 MB data range, the Python interface utilizing ctypes exhibits the shortest execution times. This outcome is somewhat unexpected, considering that the ctypes interface typically introduces overhead due to the necessity of calling C functions from Python. Therefore, it was not anticipated that the ctypes-based Python interface would demonstrate superior execution speed compared to the native C implementation.\u003c/p\u003e \u003cp\u003eThis unexpected observation may be explained by the operational workflow within the CoToCoA framework\u0026rsquo;s Worker process. As depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig9\" class=\"InternalRef\"\u003e9\u003c/span\u003e, when the pollreq function is executed by the Worker, it internally calls the MPI_Probe function to wait for incoming data from the Coupler. Accordingly, the measured execution time of the function encompasses both the waiting period and the data transmission phase. Furthermore, prior to execution of pollreq, the C program completes initialization and preparatory operations more efficiently, thereby entering the waiting state earlier. This sequence of events may lead to inflated measured runtimes for the C implementation.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eAs data size increases, the performance of the Python interface based on ctypes converges with that of the C interface. These findings indicate that ctypes provides an effective bridge between the high-level accessibility of Python and the efficiency of C, particularly for larger data volumes.\u003c/p\u003e \u003cp\u003eThe analysis presented above demonstrates that Python interfaces do not match the execution performance of the native C interface. Nevertheless, due to the architectural design of the CoToCoA framework, the observed differences in data transfer speed when utilizing the Python interface do not significantly affect the overall system operation. This robustness is primarily attributed to the Coupler\u0026rsquo;s use of MPI_Iprobe and a request queue, which together enable non-blocking reception of requests from the Requester. Consequently, the Requester is not required to wait for a Worker to complete its operation before proceeding with subsequent requests.\u003c/p\u003e \u003cp\u003eWith respect to the readarea function, the C interface consistently achieves the shortest execution times for all tested data sizes, underscoring its high efficiency for data transfer operations. The Python interface based on mpi4py exhibits longer execution times compared to C, particularly as the data size increases. In contrast, the ctypes-based Python interface demonstrates performance that closely approaches that of the C implementation across all examined data volumes.\u003c/p\u003e \u003cp\u003eThe relatively slower performance of the Python interfaces, particularly mpi4py, can be attributed to the additional abstraction layers and overhead associated with connecting Python to MPI operations. The ctypes-based approach mitigates some of this overhead by supporting direct calls to C library functions, resulting in execution times that more closely match those of the C implementation. Nonetheless, some performance penalties persist, reflecting the unavoidable costs of crossing the language boundary.\u003c/p\u003e \u003cp\u003eFor the readarea function, although differences in performance exist among the various interfaces, the use of MPI_Get enables direct access to the Requester\u0026rsquo;s memory, resulting in reduced execution times. Consequently, while the Python interfaces introduce some overhead, this does not impact the actual numerical simulation process. The function\u0026rsquo;s efficiency in data transfer ensures that use of the Python interface does not negatively affect the overall performance of the simulation.\u003c/p\u003e \u003cp\u003eWhen comparing pollreq and readarea, for each data size from 1 MB to 16 MB, the readarea function consistently exhibits shorter execution times. Specifically, the C interface demonstrates the highest performance, followed by the Python interface using ctypes, and then the Python interface using mpi4py. As the data size increases, the difference in execution time between pollreq and readarea becomes more pronounced.\u003c/p\u003e \u003cp\u003eThe pollreq function requires more execution time than readarea across all interfaces, especially as the data volume increases. This result is explained by the workflow of pollreq, which is based on MPI_Recv and obliges the Worker to wait for requests from the Requester before proceeding, thereby introducing additional latency. In contrast, readarea is built on MPI_Get, a component of MPI one-sided communication, permitting direct access to the memory of the Requester and eliminating the need to wait for a request to be sent.\u003c/p\u003e"},{"header":"6 Application to Visualization of MHD Simulations","content":"\u003cp\u003eThe Python interface developed for the CoToCoA framework has been designed for real-time visualization and machine learning applications, as well as the analysis of simulation outputs using artificial intelligence algorithms. To demonstrate the capability and practicality of the Python interface, it was applied to the visualization of results from a magnetohydrodynamics (MHD) simulation code [5]. Within this visualization framework, the MHD simulation code functions as the Requester, while a Python program serves as the Worker, performing post-processing and visualization of simulation data through the developed interface.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe architecture for visualizing MHD simulation results using CoToCoA is illustrated in Figure 10. In this structure, the Requester, implemented in Fortran, executes the MHD simulation computations. The Coupler, also written in Fortran, aggregates data from each process associated with the Requester and constructs the complete grid dataset. Subsequently, the Worker, implemented as a Python program, receives the consolidated grid data from the Coupler and performs visualization tasks.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIn this workflow, each process of the MHD simulation initiates a calculation loop within the Requester. Upon completing each computational step, the process transmits the global index corresponding to its computational domain to the Coupler. Simultaneously, the Coupler monitors for incoming data from the Requester through the MPI_Iprobe function. Upon receipt of index information, the Coupler retrieves the relevant data from the memory of the Requester process and incorporates this data into the global grid using the provided indices. Once all process data has been collected, the complete grid is assembled and placed into a queue. The Coupler then transmits the queued data to the Python-based Worker for visualization.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThis structure ensures that the calculation of each step in the MHD simulation proceeds independently, without requiring the Worker to complete its visualization task before the next step commences. Each process is responsible solely for transmitting index information, thereby optimizing communication time.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe core structure of the visualization framework is presented in Figure 11.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e6.1 Results of the Visualization\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eUtilizing this visualization framework, the MHD simulation program achieves real-time visualization without the need for file-based input and output operations. Figure 12 presents the visualization results. Each process within the MHD simulation program, acting as a Requester, transmits its local grid computation results to the Coupler after a predefined number of computational steps. The Coupler aggregates the local grid data from the memory of the Requester processes, composes the global grid, and forwards this information to the Python program. The Python program subsequently generates visualizations of the global grid using the Matplotlib library.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe MHD simulation used here has a grid size of\u0026nbsp;\u003cimg width=\"117\" height=\"19\" src=\"https://myfiles.space/user_files/58895_8739fc6c57c1c19a/58895_custom_files/img1777384451.png\" v:shapes=\"_x0000_i1025\" alt=\"image\"\u003e. If written to disk, a single output would be about 6 GB. In the test, 268,800 time steps were performed, corresponding to 210 minutes of simulated time. The MHD simulation (Requester) was coupled with Python-based visualization (Worker) using CoToCoA, and visualization was performed at every time step. The data are copied directly from the Requester\u0026rsquo;s memory to the Worker\u0026rsquo;s memory, so the simulation produces no file output. With file-based visualization, the required storage would be\u0026nbsp;\u003cimg width=\"109\" height=\"19\" src=\"https://myfiles.space/user_files/58895_8739fc6c57c1c19a/58895_custom_files/img1777384461.png\" v:shapes=\"_x0000_i1025\" alt=\"image\"\u003e\u0026nbsp;time steps (about 1,600 TB). Since current production runs are more than two orders of magnitude larger, this approach is not practical at present. CoToCoA makes it possible to visualize such runs without generating time-step files. For this problem size, one output takes about 0.08 s with distributed I/O using 448 processes (about 14 MB per process). Writing all time steps would therefore take\u0026nbsp;\u003cimg width=\"192\" height=\"19\" src=\"https://myfiles.space/user_files/58895_8739fc6c57c1c19a/58895_custom_files/img177738445139.png\" v:shapes=\"_x0000_i1025\" alt=\"image\"\u003e\u0026nbsp;(about 6 hours). This I/O time is avoided by using CoToCoA for in-situ visualization.\u003c/p\u003e\n\u003cp\u003eBeyond visualization, the described framework is structured to support direct integration with machine learning and artificial intelligence models during simulation. By substituting the visualization component of the Python Worker with an AI learning module, simulation data can be utilized for real-time training of AI models. For instance, grid data produced at each computational timestep can be harnessed to train AI models for predictive analysis or optimization. This configuration enhances the efficiency and speed of combining simulations with AI-driven learning processes. The integration obviates the requirement for intermediate file storage, thereby supporting high-frequency data analysis at every computational step.\u0026nbsp;\u003c/p\u003e"},{"header":"7 Conclusion","content":"\u003cp\u003eThis research presents the design and comprehensive assessment of a Python interface for the CoToCoA framework, advancing its applicability within contemporary computational science. Recognizing the prominent role of Python in scientific computing, data analysis, machine learning, and artificial intelligence, the development of a Python interface for CoToCoA strengthens interoperability between programming languages and merges the computational efficiency of C with Python\u0026rsquo;s robust data processing capabilities.\u003c/p\u003e \u003cp\u003eTwo principal approaches were examined in developing the Python interface: direct implementation using mpi4py and the utilization of ctypes for accessing C functions within CoToCoA from Python. This interface enables Python programs to operate as Workers, processing data transmitted from Requesters.\u003c/p\u003e \u003cp\u003ePerformance evaluations were conducted by testing the data transfer functions of CoToCoA, specifically pollreq and readarea. Results demonstrate that, for pollreq, the Python interface employing ctypes approaches the performance of the C interface as data size grows. Conversely, for readarea, the Python interface with mpi4py exhibits greater execution times compared to C, particularly with increasing data volume. The ctypes-based Python interface consistently approximates the C interface\u0026rsquo;s performance across varying data sizes.\u003c/p\u003e \u003cp\u003eThe comparative analysis of pollreq and readarea reveals that readarea consistently achieves lower execution times. This outcome is attributed to its foundation on MPI_Get, an element of MPI one-sided communication that permits direct memory access from the Requester. In contrast, pollreq is structured around MPI_Recv, which requires the Worker to await incoming requests before proceeding, introducing additional latency.\u003c/p\u003e \u003cp\u003eThe evaluation indicates that the Python interface utilizing ctypes achieves performance comparable to C, as it enables direct function calls to C libraries. This finding demonstrates that ctypes effectively bridges the gap between Python\u0026rsquo;s high-level accessibility and the low-level efficiency of C, especially with large datasets.\u003c/p\u003e \u003cp\u003eThe developed Python interface was further applied to the visualization of simulation results from a magnetohydrodynamics (MHD) simulation code, thereby demonstrating its practical utility. Through this application, the framework has shown significant potential for integration with Python-based tools, including machine learning and artificial intelligence models.\u003c/p\u003e \u003cp\u003eBy leveraging Python\u0026rsquo;s extensive data processing and visualization libraries, the interface enables the use of advanced analytical tools in simulation analysis. Furthermore, the scope of this interface is anticipated to expand to applications in deep learning and artificial intelligence.\u003c/p\u003e \u003cp\u003eBuilding upon the current visualization capabilities, future research may pursue direct transmission of simulation data to machine learning models during runtime. This approach would enable real-time interaction between simulations and ML/AI models, supporting tasks such as model training, anomaly detection, and prediction without dependence on intermediate file storage. Eliminating extensive file-based input/output operations would conserve disk space and mitigate data management overhead, thereby enhancing the speed and efficiency of integrating simulations with ML/AI methodologies.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics approval and consent to participate\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;Not applicable\u0026rdquo; in this section.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;Not applicable\u0026rdquo; in this section.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of materials\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe CoToCoA framework including the Python interface is available in the web site (https://github.com/tnanri/cotocoa) and doi:10.5281/zenodo.5655840.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare no competing interests.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis work was supported by JSPS KAKENHI Grant Number 22K12049 and 25K15146.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026apos; contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eK.F. designed the scientific content and Python interface, H.Z. implemented the Python interface. K. F. and H. Z. wrote this manuscript. T.N. contributed to theoretical support of CoToCoA and evaluation of Python interface. Y.M. and Y. K. contributed to the application of Python to scientific simulation. All authors discussed the results and contributed to the final manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis research used computational resources by Academic Center for Computing and Media Studies, Kyoto University. This study uses computational resources of the KDK computer system at the Research Institute for Sustainable Humanosphere, Kyoto University, and the HPCI system provided by the Research Institute for Information Technology, Kyushu University, and the Cyberscience Center, Tohoku University, through the HPCI System Research Project (Project IDs: hp230046, hp240018, and hp250070). This work is partly supported by the Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures (JHPCN) in Japan (Project ID: jh250014).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026apos; information\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eResearch Institute for Humanity and Nature, Kyoto, Japan\u003c/p\u003e\n\u003cp\u003eKeiichiro FUKAZAWA\u003c/p\u003e\n\u003cp\u003eGraduate School of Informatics, Kyoto University, Kyoto, Japan\u003c/p\u003e\n\u003cp\u003eHaichao ZHAO\u003c/p\u003e\n\u003cp\u003eResearch Institute for Information Technology, Kyushu University, Fukuoka, Japan\u003c/p\u003e\n\u003cp\u003eTakeshi NANRI\u003c/p\u003e\n\u003cp\u003eGraduate School of System Informatics, Kobe University, Kobe, Japan\u003c/p\u003e\n\u003cp\u003eYohei MIYAKE\u003c/p\u003e\n\u003cp\u003eGraduate School of Science Department of Geophysics, Tohoku University, Sendai, Japan\u003c/p\u003e\n\u003cp\u003eYuto Katoh\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n \u003cli\u003eJohn T. Hwang and Joaquim R.R.A. Martins. 2018. A Computational Architecture for Coupling Heterogeneous Numerical Models and Computing Coupled Derivatives. ACM Trans. Math. Softw. 44, 4, Article 37 (December 2018), 39 pages. https://doi.org/10.1145/3182393.\u003c/li\u003e\n \u003cli\u003eW. Gropp, E. Lusk, and A. Skjellum. Using MPI: portable parallel programming with the message-passing interface, vol. 1. MIT press, 1999.\u003c/li\u003e\n \u003cli\u003eK. Fukazawa, Y. Katoh, T. Nanri, and Y. Miyake. Application of cross-reference framework cotocoa to macro- and micro-scale simulations of planetary magnetospheres. In 2019 Seventh International Symposium on Computing and Networking Workshops (CANDARW), pp. 121--124, 2019.\u003c/li\u003e\n \u003cli\u003eT. Nanri. Code-to-code adapter. https://github.com/tnanri/cotocoa,2024. Accessed: 2024-12-23.\u003c/li\u003e\n \u003cli\u003eH. Bungartz, F. Lindner, B. Gatzhammer, M. Mehl, K. Scheufele, A. Shukaev, and B. Uekermann. precice-a fully parallel library for multi-physics surface coupling. Computers \\\u0026amp; Fluids, vol. 141, pp. 250-258, 2016.\u003c/li\u003e\n \u003cli\u003eD. R Gaston, C. J Permann, J. W Peterson, A. E Slaughter, D. Andres, Y. Wang, M. P Short, D. M Perez, M. R Tonks, and J. Ortensi. Physics-based multiscale coupling for full core nuclear reactor simulation. Annals of Nuclear Energy, vol. 84, pp. 45--54, 2015.\u003c/li\u003e\n \u003cli\u003eD. M Beazley. Swig: An easy to use tool for integrating scripting languages with c and c++. In Tcl/Tk Workshop, vol. 43, pp. 74, 1996.\u003c/li\u003e\n \u003cli\u003eK. Fukazawa, T. Ogino, and R. J Walker. Configuration and dynamics of the Jovian magnetosphere. Journal of Geophysical Research: Space Physics, 111(A10), 2006.\u003c/li\u003e\n \u003cli\u003eY. Miyake, Y. Sunada, Y. Tanaka, K. Nakazawa, T. Nanri, K. Fukazawa, and Y. Katoh. Implementation of coupled numerical analysis of magnetospheric dynamics and spacecraft charging phenomena via code-to-code adapter (cotocoa) framework. In International Conference on Computational Science, pp. 438--452. Springer, 2023.\u003c/li\u003e\n \u003cli\u003eSystem Specification of Supercomputer at Kyoto University (2023) (https://www.iimc.kyoto-u.ac.jp/en/services/comp/supercomputer/system/specification)\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Table","content":"\u003cp\u003eTable 1 is available in the Supplementary Files section.\u003c/p\u003e\n"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"earth-planets-and-space","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"epsp","sideBox":"Learn more about [Earth, Planets and Space](http://earth-planets-space.springeropen.com)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/epsp/default.aspx","title":"Earth, Planets and Space","twitterHandle":"@SpringerOpen","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Coupled Computation, Cross-reference Simulation, Python Interface, Real-time visualization, Real-time AI training","lastPublishedDoi":"10.21203/rs.3.rs-9195901/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9195901/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eCross-reference simulation is a calculation model combining multiple simulation codes when some simulation codes need the data in other simulation codes to do their own calculation. To provide an efficient and convenient solution for cross-reference Solar-Terrestrial Physics (STP) simulation, a framework called Code-To-Code Adapter (CoToCoA) Library has been developed. CoToCoA was designed to implement cross-reference simulation with minimal modifications to the simulation codes while maintaining low overhead and currently supports only C and Fortran. To further enhance its usability by integrating CoToCoA with Python, the development of a Python interface is performed. Two development approaches were conducted: (1) direct development of CoToCoA in Python and (2) the use of a library to call C functions of CoToCoA from Python. To evaluate the performance of the Python interfaces, experiments of data transfer methods in CoToCoA are performed. As a result of the evaluation, using the library implementation has a close performance to C, while the direct implementation needs more additional time. The results show that the Python interface using the library enables efficient data transfer from both C and Fortran-based STP simulation programs to Python for advanced data analysis and processing. Furthermore, the developed Python interface was applied to visualize magnetohydrodynamics simulation results in real-time. Based on the application, the framework has shown the potential to enable direct integration with other Python tools, such as machine learning or artificial intelligence models.\u003c/p\u003e","manuscriptTitle":"Development of a Python Interface for a Framework of Cross-reference Solar-Terrestrial Physics Simulation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-28 13:55:15","doi":"10.21203/rs.3.rs-9195901/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewerAgreed","content":"","date":"2026-04-20T09:17:32+00:00","index":0,"fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-20T05:05:58+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-04-10T07:09:47+00:00","index":"","fulltext":""},{"type":"submitted","content":"Earth, Planets and Space","date":"2026-04-08T03:52:47+00:00","index":"","fulltext":""},{"type":"decision","content":"Major Revision","date":"2026-04-03T00:53:50+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"earth-planets-and-space","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"epsp","sideBox":"Learn more about [Earth, Planets and Space](http://earth-planets-space.springeropen.com)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/epsp/default.aspx","title":"Earth, Planets and Space","twitterHandle":"@SpringerOpen","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e3261894-0856-4d57-a0b3-337bbdab2d11","owner":[],"postedDate":"April 28th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-05-19T10:30:39+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-28 13:55:15","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9195901","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9195901","identity":"rs-9195901","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}