Introduction
Open Compute Stack (OpenCS) is a framework for modelling of large scale ODE/DAE systems, parallel evaluation of model equations and parallel simulations on shared and distributed memory systems.
The framework provides a platformindependent binary interface for modelexchange with the data structures to describe, store in computer memory and evaluate large scale ODE/DAE systems of equations. This approach differs from the typical modelexchange/cosimulation interfaces in that it does not require a human or a machine readable model definition as in modelling and modelexchange languages (i.e. Modelica, gPROMS and CellML) nor a binary interface (C API) implemented in shared libraries (i.e. Simulink and Functional Mockup Interface). For instance, in the OpenCS framework model equations are specified in the Reverse Polish (postfix) notation as an array of binary data (a Compute Stack) for direct evaluation by simulators on all platforms/operating systems (including heterogeneous systems) with no additional processing nor compilation steps. Therefore, the same modelspecification can be used on any computing platform.
API and libraries
The framework includes an API and libraries for:
 Model specification
 Direct implementation in C++
 Export from simulatorspecific data structures
 Parallel evaluation of model equations
 The OpenMP API on general purpose processors (multicore CPUs, Xeon Phi)
 The OpenCL framework on streaming processors (GPU, FPGA) and heterogeneous systems (CPU+GPU, CPU+FPGA)
 Model exchange
 The models are specified using the OpenCS API and stored as files in a platformindependent binary format (one set of files per processing element)
 The OpenCS API is used for loading the models into a host simulator and as a common interface to the data required for integration in time by ODE/DAE solvers (i.e. evaluation of equations and derivatives)
 Simulation on shared memory systems
 Embedded into a thirdparty simulator using the OpenCS API
 Using the standalone ODE/DAE simulators
 Simulation on distributed memory systems
 Embedded into a thirdparty simulator using the OpenCS API
 Using the standalone ODE/DAE simulators
Use case scenarios
Typical use case scenarios include:
 Development of custom largescale models in C++
 Parallel evaluation of model equations (i.e. in simulators without a support for parallel evaluation)
 Universal parallel simulations on shared and distributed memory systems
 Export of existing models from thirdparty simulators for:
 Modelexchange
 Benchmarks between simulators and ODE/DAE solvers
 Benchmarks between HPC systems (since the common modelspecification is used on all platforms), i.e. between various heterogeneous CPU+GPU and CPU+FPGA clusters
The advantages/benefits
The OpenCS framework offers the numerous benefits:
 A single software is used for numerical solution of any system of differential and algebraic equations (ODE or DAE) of any size and on all platforms
 The model specification contains only the lowlevel model description and therefore can be generated from any modelling software
 The model specification data structures are stored as files in a platformindependent binary format and used as inputs for parallel simulations on all platforms
 Model equations are specified in a platform and programming language independent fashion as an array of binary data (Compute Stacks)
 Equations of any type (differential or algebraic) and any size are supported and can be evaluated on virtually all computing devices (including heterogeneous systems)
 Evaluation of equations on different computing platforms is straightforward (an input parameter)
The OpenCS methodology
The framework is based on the methodology for parallel numerical solution of general systems of nonlinear differential and algebraic equations on shared and distributed memory systems presented in the following articles:
 Parallelisation of equationbased simulation programs on heterogeneous computing systems (Nikolić, 2018a).
 Parallelisation of equationbased simulation programs on distributed memory systems (Nikolić, 2018b).
The methodology includes the following components:
 An algorithm for transformation of model equations into a data structure suitable for parallel evaluation
 Data structures for model specification that contain all information required for numerical solution such as:
 the model structure
 the model equations
 the sparsity pattern
 partition data
 An algorithm for partitioning of general systems of systems using multiple balancing constraints
 An algorithm for interprocess data exchange
 The simulation software for integration of general ODE/DAE systems in time
The Key Concepts
Compute Stack  The Reverse Polish (postfix) notation expression stack used as a platform and programming language independent method to describe, store in computer memory and evaluate equations of any type and any size (Nikolić, 2018a). Equations can be linear or nonlinear, algebraic or differential. Each mathematical operation and its operands are described by a specially designed csComputeStackItem_t data structure, and every equation is transformed into an array of these structures (a Compute Stack). 
Compute Stack Machine  A stack machine used to evaluate a single equation (that is a single Compute Stack) using Last In First Out (LIFO) queues. 
Compute Stack Evaluator  An interface for parallel evaluation of systems of equations (csComputeStackEvaluator_t class).
Two implementations are available (Nikolić, 2018a):

Compute Stack Model  Data structure that holds the model specification  all information required for the numerical solution,
either sequentially or in parallel (csModel_t data structure).
For sequential simulations the system is described by a single csModel_t object.
For parallel simulations the system is described by an array of csModel_t objects each holding information
about one ODE/DAE subsystem.
Every model contains the following data:

Compute Stack Differential Equations Model  A common interface that provides an API required by ODE/DAE solvers for integration of systems of differential equations in time (csDifferentialEquationModel_t class). It is derived from csModel_t class and provides functions for loading the model from input files, retrieving the sparsity pattern of the ODE/DAE system, setting the variable values/derivatives, exchanging the adjacent variables among the processing elements using the MPI interface, and evaluating equations and derivatives. 
Compute Stack Simulator  Software for sequential and parallel simulation of general ODE/DAE systems in time
(csSimulator_ODE and csSimulator_DAE, respectively).
Simulation inputs are specified in a platform and programming language independent fashion

Compute Stack Model Builder  A common interface for creation of ODE/DAE Compute Stack models.
The major use is for export of models from different simulators into the Compute Stack models.
However, it can also be directly used for model specification as in one of the suites for scientific applications.
It includes the following functionality:

Libraries and software provided
The key concepts of the OpenCS framework are implemented in the following libraries:
 cs_machine.h (headeronly Compute Stack Machine implementation in C99)
 libOpenCS_Evaluators (sequential, OpenMP and OpenCL Compute Stack Evaluator implementations)
 libOpenCS_Models (Compute Stack Model, Compute Stack Differential Equations Model and Compute Stack Model Builder implementations)
 libOpenCS_Simulators (implementations of Compute Stack ODE/DAE Simulators)
 csSimulator_ODE
 csSimulator_DAE
Dependencies
The OpenCS framework utilises the following APIs/frameworks:
numerical solvers: and libraries:Background
In general, the model specification for either sequential or parallel simulations are developed using:
 generalpurpose programming languages such as C/C++ or Fortran and one of available suites for scientific applications such as SUNDIALS, Trilinos and PETSC
 modelling languages such as Ascend, APMonitor, gPROMS and Modelica (Dymola, JModelica and OpenModelica)
 multiparadigm numerical languages such as Matlab, Scilab, Mathematica and Maple
 higherlevel fourthgeneration languages (i.e. Python) such as Assimulo and DAE Tools
 libraries for Finite Element Analysis (FEA) and Computational Fluid Dynamics (CFD) such as deal.II, libMesh and OpenFOAM
 Computer Aided Engineering (CAE) software for Finite Element Analysis and Computational Fluid Dynamics such as HyperWorks, STARCCM+/STARCD, COMSOL Multiphysics, ANSYS Fluent/CFX and Abaqus
A detailed discussion of capabilities and limitations of the available approaches for specification of model equations and development of largescale simulation programs are given in Nikolić (2016) and Nikolić (2018a).
The main idea in the OpenCS approach is to separate (simulatordependent) generation of a system of equations (i.e. meshing, discretisation and system assembly) and partitioning of the system, typically performed only once, from its parallel (in general, simulatorindependent) numerical solution which is computationally the most intensive task. While generation of the system of equations can be performed in different ways depending on the type of the problem and the method applied by a simulator, the numerical solution procedure requires only a lowlevel model description. For instance, the model description can be built using a modelling language or a CAE software utilising various discretisation methods. However, information required by ODE/DAE solvers are essentially identical: the data about the number of variables, their names, types, absolute tolerances and initial conditions, the functions for evaluation of residuals and derivatives, and the function for exchange of adjacent variables between processing elements (PEs). Therefore, in the OpenCS approach, the model specification data structures contain only the lowlevel information directly required by solvers.
A generic simulation software has been developed to utilise such a model specification:
 On shared memory systems simulations are executed on a single processing element utilising the available computing hardware (i.e. multicore CPU, GPU or heterogeneous CPU+GPU):
 On distributed memory systems simulations are executed on a number of processing elements where every processing element integrates one part (subsystem) of the overall ODE/DAE system in time and performs an interprocess communication to exchange the data between processing elements:
This way, the OpenCS framework provides a common interface required by ODE/DAE solvers for integration in time that utilises very diverse types of computing devices for parallel evaluation of model equations. The OpenCS models can be developed in C++ or exported from simulators using the provided Model Builder API.