The auto-detection wasn't working because the
CLI was always setting PYTHON_VERSIONS environment variable to all supported
versions when --python-versions was not provided.
Changes:
- click_helpers.py: Return None instead of all versions when option not provided
- helpers.py: Only set PYTHON_VERSIONS env var when explicitly specified
- main.py: Update type hints to allow None for python_versions parameter
Now the flow works correctly:
1. No --python-versions CLI option -> python_versions = None
2. None passed to Docker -> no PYTHON_VERSIONS env var set
3. Container auto-detects installed versions using pyenv
4. Tests run only on detected versions
When no --python-versions is specified at
runtime, auto-detect which Python versions are actually installed in the
container instead of defaulting to all versions from python_versions.json.
This resolves the issue where:
1. Building with --python-versions=3.12 installs only Python 3.12
2. Running without --python-versions would try to test all versions from JSON
3. Tests would fail for non-installed versions (3.8, 3.9, 3.10, 3.11, 3.13)
Now the runtime logic:
1. Uses `pyenv versions --bare` to detect installed versions
2. Maps full versions (3.12.9) to short versions (3.12)
3. Uses detected versions as default for testing
4. Falls back to JSON versions if detection fails
Example behavior:
- Build: --python-versions=3.12 (installs only 3.12.9)
- Runtime: auto-detects 3.12, tests only that version
- Explicit: --python-versions=3.11,3.12 still works as before
Adds a new CLI option to control test execution behavior when failures occur:
- Without --stop-on-error (default): Continue running all tests even if some fail
- With --stop-on-error: Stop execution on first test failure (previous behavior)
This enables comprehensive debugging by running both opm-common and omp-simulators
tests in a single run, showing all failures at once rather than stopping at the first one.
Changes:
- Add --stop-on-error flag to run-docker-image command
- Pass STOP_ON_ERROR environment variable to Docker container
- Update entrypoint.py to handle continue-on-error behavior
- Modify test functions to check return codes and continue when appropriate
- Maintain backward compatibility with existing error handling
This enhancement allows specifying which Python versions to install during
Docker image builds, significantly reducing build times when only testing
specific Python versions during debugging.
Changes:
- Add --python-versions option to build-docker-image command
- Pass BUILD_PYTHON_VERSIONS build argument to Docker
- Update install_python_versions.py to respect BUILD_PYTHON_VERSIONS env var
- Validate requested versions against supported versions in python_versions.json
- Maintain backward compatibility (installs all versions if not specified)
Usage examples:
# Build with only Python 3.12 (much faster for debugging)
opm-wheels build-docker-image --docker-os="ubuntu-24.04" --python-versions="3.12"
# Build with multiple specific versions
omp-wheels build-docker-image --python-versions="3.11,3.12"
# Build with all versions (original behavior)
opm-wheels build-docker-image
Benefits:
- Reduces build time from ~7 minutes to ~2 minutes for single Python version
- Enables faster iteration during debugging of Docker-specific issues
- Particularly valuable when combined with --test-cases-* for focused testing
This enhancement adds the ability to run specific test cases in the Docker
testing infrastructure, enabling more focused debugging and faster iteration
when investigating test failures.
Changes:
- Add --test-cases-common and --test-cases-simulators CLI options to opm-wheels
- Pass test selection as environment variables to Docker container
- Update entrypoint.py to support selective test execution:
* Run only opm-common tests when --test-cases-common is specified
* Run only opm-simulators tests when --test-cases-simulators is specified
* Maintain backward compatibility (run all tests if no selection specified)
- Support both host-tests-dir and cloned repo testing modes
Usage examples:
# Run only esmry tests from opm-common
opm-wheels run-docker-image --test-cases-common=esmry
# Run only basic tests from opm-simulators
opm-wheels run-docker-image --test-cases-simulators=basic
# Run specific tests from both suites
opm-wheels run-docker-image --test-cases-common=esmry --test-cases-simulators=mpi
This enables efficient debugging of Docker-specific test failures by isolating
problematic test cases without running the full test suite.
- Export PIP_ROOT_USER_ACTION=ignore to silence root-user warning in manylinux.
- Remove pytest-runner from build-time pip installs; no longer needed after
removing setup_requires.
- Replace direct setup.py invocations with: python -m build --sdist --wheel
- Add comments explaining motivation (setuptools deprecations, build isolation)
and why we drop explicit --plat-name/--python-tag (auditwheel/env handles).
- Ensure the build frontend is present per-interpreter by installing 'build'.
Also add build-backend to pyproject.toml with notes and TODO for full modernization.
Copy host directory files from opm-simulators/python and
opm-common/python to container to avoid conflict with
opm-simulators/python/opm/ and opm-common/python/opm/ folders when
running "python -m unittest discover" from the python directories.
An alternative would be to rename the opm directories in the host.
Reorganize Python folder: separate Docker/PyPI from CMake/PyBind11 workflows
and implement comprehensive build automation
This commit represents a comprehensive restructuring and enhancement of the
OPM Python build system, addressing maintainability, consistency, and
usability issues through architectural improvements and automation.
FILES CHANGED
=============
Refactored:
- python/Dockerfile -> python/docker/Dockerfile
- python/generate-pypi-package.sh -> python/docker/scripts/generate-pypi-package.sh
- python/setup-docker-image.sh -> python/docker/scripts/setup-docker-image.sh
- python/build-deps.sh -> python/docker/scripts/build-boost.sh
- python/build-deps.sh -> python/docker/scripts/build-dune.sh
Added (24 files):
- python/docker/python_versions.json (single source of truth)
- python/docker/scripts/sync_versions.sh
- python/docker/scripts/read_python_config.py
- python/docker/run-docker-build.sh
- python/docker/README.md
- python/docker/test_wheels/* (complete testing framework)
Modified:
- python/setup.py.in
- .gitignore
MAJOR CHANGES
=============
Architecture Restructure
------------------------
- Separated build workflows: Clearly distinguished Docker/PyPI wheel
generation from CMake/PyBind11 compilation
- Moved Docker-specific files: Relocated docker build system to
appropriate locations
- Streamlined organization: Created logical separation between different
build contexts
Single Source of Truth Configuration
-------------------------------------
- Centralized Python version management: Created python_versions.json as
authoritative configuration
- Automated synchronization: Built sync_versions.sh script to maintain
consistency across all components
- Dynamic configuration: Components now read from JSON rather than
hardcoded values
Enhanced Docker Build System
----------------------------
- Multi-OS support: Added Ubuntu 22.04, Ubuntu 24.04, and Debian 11
Dockerfiles
- Improved caching: Split dependencies into separate Docker layers for
better build performance
- Fixed PyPI wheel generation: Added LD_LIBRARY_PATH setup for proper
library bundling, corrected auditwheel wheel copying from wheelhouse/
instead of dist/
- Optimized Docker layer structure: Delayed source code copying until
final stage, preventing dependency rebuilds on source changes and
reducing iteration time from 30+ to <5 minutes during development
Build Automation & Tooling
--------------------------
- Comprehensive wheel testing: Created opm-wheels CLI tool for building
and testing Docker images
- Automated validation: Integrated Python version validation and error
handling
- Enhanced scripts: Improved generate-pypi-package.sh with better error
handling and version management
Testing Framework
-----------------
- Docker-based testing: Complete framework for testing wheels across
multiple Python versions and OS configurations
- Automated validation: Wheel installation and import testing with
comprehensive error reporting
- Multi-version support: Parallel testing across Python 3.8-3.13
- Platform verification: Validates manylinux wheels install and function
correctly across Ubuntu 22.04/24.04 and Debian 11
Documentation & Usability
-------------------------
- Comprehensive maintenance guides: Detailed instructions for
adding/removing Python versions
- Clear workflows: Step-by-step procedures for build system maintenance
- Improved CLI experience: Better error messages, path handling, and
user guidance
- Updated documentation: Enhanced README files with practical examples
and troubleshooting
IMPACT
======
This restructure transforms the Python build system from a collection of
loosely coupled scripts into a cohesive, maintainable, and reliable build
automation system. The single source of truth configuration eliminates the
primary source of maintenance overhead, while the enhanced testing
framework provides confidence in the quality of generated wheels.
The changes enable the OPM project to more easily support new Python
versions, maintain consistency across platforms, and provide a better
developer experience for both maintainers and users.
David Landa Marban
Håkon Hægland
Arne Morten Kvarving