Extend the Python blackoil simulator to support two-phase gas-water
simulations by extracting common functionality into a template base
class and implementing proper module separation.
- Extract PyBaseSimulator<TypeTag> template base class containing all
common simulator functionality (20+ methods including advance, run,
step, property access, and MPI handling)
- Add PyGasWaterSimulator for two-phase gas-water simulations using
FlowGasWaterProblem TypeTag with oil phase disabled
- Split Python modules to avoid ODR violations:
- BlackOil.so for PyBlackOilSimulator (FlowProblemTPFA)
- GasWater.so for PyGasWaterSimulator (FlowGasWaterProblem)
- Implement proper template instantiation control with explicit
instantiations per module
- Eliminate ~400 lines of code duplication between simulators
- Clean inheritance hierarchy: derived classes contain only constructors
- Type-safe specialization through OPM's TypeTag property system
- Separate docstring generation for each simulator type
- Independent build targets with shared infrastructure
- Update CMakeLists.txt to support multiple pybind11 modules
- Add docstrings_gw_simulators.json for gas-water simulator documentation
- Remove centralized Pybind11Exporter.cpp in favor of distributed exports
- Implement proper build dependencies and custom targets
This establishes a scalable foundation for extending OPM's Python
bindings to support additional simulator types (thermal, polymer,
solvent) while maintaining type safety and avoiding symbol conflicts.
Files changed: 21 files, 915 insertions, 428 deletions
- Export PIP_ROOT_USER_ACTION=ignore to silence root-user warning in manylinux.
- Remove pytest-runner from build-time pip installs; no longer needed after
removing setup_requires.
- Replace direct setup.py invocations with: python -m build --sdist --wheel
- Add comments explaining motivation (setuptools deprecations, build isolation)
and why we drop explicit --plat-name/--python-tag (auditwheel/env handles).
- Ensure the build frontend is present per-interpreter by installing 'build'.
Also add build-backend to pyproject.toml with notes and TODO for full modernization.
- Drop legacy setup_requires and test_suite to stop
setuptools _DeprecatedInstaller / fetch_build_eggs and
unknown distribution option warnings.
- Add inline comment explaining rationale and PEP 517 context.
Copy host directory files from opm-simulators/python and
opm-common/python to container to avoid conflict with
opm-simulators/python/opm/ and opm-common/python/opm/ folders when
running "python -m unittest discover" from the python directories.
An alternative would be to rename the opm directories in the host.
Reorganize Python folder: separate Docker/PyPI from CMake/PyBind11 workflows
and implement comprehensive build automation
This commit represents a comprehensive restructuring and enhancement of the
OPM Python build system, addressing maintainability, consistency, and
usability issues through architectural improvements and automation.
FILES CHANGED
=============
Refactored:
- python/Dockerfile -> python/docker/Dockerfile
- python/generate-pypi-package.sh -> python/docker/scripts/generate-pypi-package.sh
- python/setup-docker-image.sh -> python/docker/scripts/setup-docker-image.sh
- python/build-deps.sh -> python/docker/scripts/build-boost.sh
- python/build-deps.sh -> python/docker/scripts/build-dune.sh
Added (24 files):
- python/docker/python_versions.json (single source of truth)
- python/docker/scripts/sync_versions.sh
- python/docker/scripts/read_python_config.py
- python/docker/run-docker-build.sh
- python/docker/README.md
- python/docker/test_wheels/* (complete testing framework)
Modified:
- python/setup.py.in
- .gitignore
MAJOR CHANGES
=============
Architecture Restructure
------------------------
- Separated build workflows: Clearly distinguished Docker/PyPI wheel
generation from CMake/PyBind11 compilation
- Moved Docker-specific files: Relocated docker build system to
appropriate locations
- Streamlined organization: Created logical separation between different
build contexts
Single Source of Truth Configuration
-------------------------------------
- Centralized Python version management: Created python_versions.json as
authoritative configuration
- Automated synchronization: Built sync_versions.sh script to maintain
consistency across all components
- Dynamic configuration: Components now read from JSON rather than
hardcoded values
Enhanced Docker Build System
----------------------------
- Multi-OS support: Added Ubuntu 22.04, Ubuntu 24.04, and Debian 11
Dockerfiles
- Improved caching: Split dependencies into separate Docker layers for
better build performance
- Fixed PyPI wheel generation: Added LD_LIBRARY_PATH setup for proper
library bundling, corrected auditwheel wheel copying from wheelhouse/
instead of dist/
- Optimized Docker layer structure: Delayed source code copying until
final stage, preventing dependency rebuilds on source changes and
reducing iteration time from 30+ to <5 minutes during development
Build Automation & Tooling
--------------------------
- Comprehensive wheel testing: Created opm-wheels CLI tool for building
and testing Docker images
- Automated validation: Integrated Python version validation and error
handling
- Enhanced scripts: Improved generate-pypi-package.sh with better error
handling and version management
Testing Framework
-----------------
- Docker-based testing: Complete framework for testing wheels across
multiple Python versions and OS configurations
- Automated validation: Wheel installation and import testing with
comprehensive error reporting
- Multi-version support: Parallel testing across Python 3.8-3.13
- Platform verification: Validates manylinux wheels install and function
correctly across Ubuntu 22.04/24.04 and Debian 11
Documentation & Usability
-------------------------
- Comprehensive maintenance guides: Detailed instructions for
adding/removing Python versions
- Clear workflows: Step-by-step procedures for build system maintenance
- Improved CLI experience: Better error messages, path handling, and
user guidance
- Updated documentation: Enhanced README files with practical examples
and troubleshooting
IMPACT
======
This restructure transforms the Python build system from a collection of
loosely coupled scripts into a cohesive, maintainable, and reliable build
automation system. The single source of truth configuration eliminates the
primary source of maintenance overhead, while the enhanced testing
framework provides confidence in the quality of generated wheels.
The changes enable the OPM project to more easily support new Python
versions, maintain consistency across platforms, and provide a better
developer experience for both maintainers and users.
These were originally obtained through transitive includes,
especially from Main.hpp, but that's not a stable condition if the
set of transitive includes changes.
With this keyword, the SPE1CASE1.UNRST file could not be read in correctly at some point, until I've added a check in opm-models that checks this, this went unnoticed probably.
Håkon Hægland
Svenn Tveit
Arne Morten Kvarving
Bård Skaflestad
Lisa Julia Nebel
Markus Blatt
hnil
Tor Harald Sandve