Open Compute Stack (OpenCS) is a framework for modelling of large scale ODE/DAE systems, parallel evaluation of model equations and parallel simulations on shared and distributed memory systems.
The framework provides a platform-independent binary interface for model-exchange with the data structures to describe, store in computer memory and evaluate large scale ODE/DAE systems of equations. This approach differs from the typical model-exchange/co-simulation interfaces in that it does not require a human or a machine readable model definition as in modelling and model-exchange languages (i.e. Modelica, gPROMS and CellML) nor a binary interface (C API) implemented in shared libraries (i.e. Simulink and Functional Mock-up Interface). For instance, in the OpenCS framework model equations are specified in the Reverse Polish (postfix) notation as an array of binary data (a Compute Stack) for direct evaluation by simulators on all platforms/operating systems (including heterogeneous systems) with no additional processing nor compilation steps. Therefore, the same model-specification can be used on any computing platform.
API and libraries
The framework includes an API and libraries for:
- Model specification
- Direct implementation in C++
- Export from simulator-specific data structures
- Parallel evaluation of model equations
- The OpenMP API on general purpose processors (multi-core CPUs, Xeon Phi)
- The OpenCL framework on streaming processors (GPU, FPGA) and heterogeneous systems (CPU+GPU, CPU+FPGA)
- Model exchange
- The models are specified using the OpenCS API and stored as files in a platform-independent binary format (one set of files per processing element)
- The OpenCS API is used for loading the models into a host simulator and as a common interface to the data required for integration in time by ODE/DAE solvers (i.e. evaluation of equations and derivatives)
- Simulation on shared memory systems
- Embedded into a third-party simulator using the OpenCS API
- Using the standalone ODE/DAE simulators
- Simulation on distributed memory systems
- Embedded into a third-party simulator using the OpenCS API
- Using the standalone ODE/DAE simulators
Use case scenarios
Typical use case scenarios include:
- Development of custom large-scale models in C++
- Parallel evaluation of model equations (i.e. in simulators without a support for parallel evaluation)
- Universal parallel simulations on shared and distributed memory systems
- Export of existing models from third-party simulators for:
- Benchmarks between simulators and ODE/DAE solvers
- Benchmarks between HPC systems (since the common model-specification is used on all platforms), i.e. between various heterogeneous CPU+GPU and CPU+FPGA clusters
The OpenCS framework offers the numerous benefits:
- A single software is used for numerical solution of any system of differential and algebraic equations (ODE or DAE) of any size and on all platforms
- The model specification contains only the low-level model description and therefore can be generated from any modelling software
- The model specification data structures are stored as files in a platform-independent binary format and used as inputs for parallel simulations on all platforms
- Model equations are specified in a platform and programming language independent fashion as an array of binary data (Compute Stacks)
- Equations of any type (differential or algebraic) and any size are supported and can be evaluated on virtually all computing devices (including heterogeneous systems)
- Evaluation of equations on different computing platforms is straightforward (an input parameter)
The OpenCS methodology
The framework is based on the methodology for parallel numerical solution of general systems of non-linear differential and algebraic equations on shared and distributed memory systems presented in the following articles:
- Parallelisation of equation-based simulation programs on heterogeneous computing systems (Nikolić, 2018a).
- Parallelisation of equation-based simulation programs on distributed memory systems (Nikolić, 2018b).
The methodology includes the following components:
- An algorithm for transformation of model equations into a data structure suitable for parallel evaluation
- Data structures for model specification that contain all information required for numerical solution such as:
- the model structure
- the model equations
- the sparsity pattern
- partition data
- An algorithm for partitioning of general systems of systems using multiple balancing constraints
- An algorithm for inter-process data exchange
- The simulation software for integration of general ODE/DAE systems in time
The Key Concepts
|Compute Stack||The Reverse Polish (postfix) notation expression stack used as a platform and programming language independent method to describe, store in computer memory and evaluate equations of any type and any size (Nikolić, 2018a). Equations can be linear or non-linear, algebraic or differential. Each mathematical operation and its operands are described by a specially designed csComputeStackItem_t data structure, and every equation is transformed into an array of these structures (a Compute Stack).|
|Compute Stack Machine||A stack machine used to evaluate a single equation (that is a single Compute Stack) using Last In First Out (LIFO) queues.|
|Compute Stack Evaluator||An interface for parallel evaluation of systems of equations (csComputeStackEvaluator_t class).
Two implementations are available (Nikolić, 2018a):
|Compute Stack Model||Data structure that holds the model specification - all information required for the numerical solution,
either sequentially or in parallel (csModel_t data structure).
For sequential simulations the system is described by a single csModel_t object.
For parallel simulations the system is described by an array of csModel_t objects each holding information
about one ODE/DAE sub-system.
Every model contains the following data:
|Compute Stack Differential Equations Model||A common interface that provides an API required by ODE/DAE solvers for integration of systems of differential equations in time (csDifferentialEquationModel_t class). It is derived from csModel_t class and provides functions for loading the model from input files, retrieving the sparsity pattern of the ODE/DAE system, setting the variable values/derivatives, exchanging the adjacent variables among the processing elements using the MPI interface, and evaluating equations and derivatives.|
|Compute Stack Simulator||Software for sequential and parallel simulation of general ODE/DAE systems in time
(csSimulator_ODE and csSimulator_DAE, respectively).
Simulation inputs are specified in a platform and programming language independent fashion
|Compute Stack Model Builder||A common interface for creation of ODE/DAE Compute Stack models.
The major use is for export of models from different simulators into the Compute Stack models.
However, it can also be directly used for model specification as in one of the suites for scientific applications.
It includes the following functionality:
Libraries and software provided
The key concepts of the OpenCS framework are implemented in the following libraries:
- cs_machine.h (header-only Compute Stack Machine implementation in C99)
- libOpenCS_Evaluators (sequential, OpenMP and OpenCL Compute Stack Evaluator implementations)
- libOpenCS_Models (Compute Stack Model, Compute Stack Differential Equations Model and Compute Stack Model Builder implementations)
- libOpenCS_Simulators (implementations of Compute Stack ODE/DAE Simulators)
The OpenCS framework utilises the following APIs/frameworks:
In general, the model specification for either sequential or parallel simulations are developed using:
- general-purpose programming languages such as C/C++ or Fortran and one of available suites for scientific applications such as SUNDIALS, Trilinos and PETSC
- modelling languages such as Ascend, APMonitor, gPROMS and Modelica (Dymola, JModelica and OpenModelica)
- multi-paradigm numerical languages such as Matlab, Scilab, Mathematica and Maple
- higher-level fourth-generation languages (i.e. Python) such as Assimulo and DAE Tools
- libraries for Finite Element Analysis (FEA) and Computational Fluid Dynamics (CFD) such as deal.II, libMesh and OpenFOAM
- Computer Aided Engineering (CAE) software for Finite Element Analysis and Computational Fluid Dynamics such as HyperWorks, STAR-CCM+/STAR-CD, COMSOL Multiphysics, ANSYS Fluent/CFX and Abaqus
A detailed discussion of capabilities and limitations of the available approaches for specification of model equations and development of large-scale simulation programs are given in Nikolić (2016) and Nikolić (2018a).
The main idea in the OpenCS approach is to separate (simulator-dependent) generation of a system of equations (i.e. meshing, discretisation and system assembly) and partitioning of the system, typically performed only once, from its parallel (in general, simulator-independent) numerical solution which is computationally the most intensive task. While generation of the system of equations can be performed in different ways depending on the type of the problem and the method applied by a simulator, the numerical solution procedure requires only a low-level model description. For instance, the model description can be built using a modelling language or a CAE software utilising various discretisation methods. However, information required by ODE/DAE solvers are essentially identical: the data about the number of variables, their names, types, absolute tolerances and initial conditions, the functions for evaluation of residuals and derivatives, and the function for exchange of adjacent variables between processing elements (PEs). Therefore, in the OpenCS approach, the model specification data structures contain only the low-level information directly required by solvers.
A generic simulation software has been developed to utilise such a model specification:
- On shared memory systems simulations are executed on a single processing element utilising the available computing hardware (i.e. multi-core CPU, GPU or heterogeneous CPU+GPU):
- On distributed memory systems simulations are executed on a number of processing elements where every processing element integrates one part (sub-system) of the overall ODE/DAE system in time and performs an inter-process communication to exchange the data between processing elements:
This way, the OpenCS framework provides a common interface required by ODE/DAE solvers for integration in time that utilises very diverse types of computing devices for parallel evaluation of model equations. The OpenCS models can be developed in C++ or exported from simulators using the provided Model Builder API.