Files
opm-simulators/python/docker

Docker Build System for OPM Python Wheels

This directory contains the Docker-based build system for generating PyPI-compatible wheels for the OPM Python bindings.

Directory Structure

docker/
├── Dockerfile                 # Main Docker build configuration
├── run-docker-build.sh        # User-facing helper script for building wheels
├── python_versions.json       # Python version definitions
├── README.md                  # This file (documentation)
├── scripts/                   # Internal build scripts
│   ├── generate-pypi-package.sh    # Wheel generation script
│   ├── setup-docker-image.sh       # Docker environment setup
│   ├── build-boost.sh             # Boost dependency build
│   ├── build-dune.sh              # DUNE dependency build
│   ├── build-deps.sh              # Additional dependencies
│   └── read_python_config.py      # Python version config reader
├── test_wheels/               # Wheel testing framework
└── [output directories]       # Generated by --output parameter (user-specified names)

Building Wheels

Use the provided helper script for simplified building with automatic logging:

# Static build with wheel extraction
./docker/run-docker-build.sh static --output wheelhouse
# Creates: docker/wheelhouse/ with wheels

# Shared libraries build
./docker/run-docker-build.sh shared --libtype shared --output wheelhouse-shared
# Creates: docker/wheelhouse-shared/ with wheels

# Build specific Python versions
./docker/run-docker-build.sh py312 --python-versions "3.12" --output test-wheels
# Creates: docker/test-wheels/ with wheels

# Debug build with detailed output (no wheel extraction)
./docker/run-docker-build.sh debug --libtype shared --no-buildx --jobs 8

# Absolute path output (advanced usage)
./docker/run-docker-build.sh release --output /tmp/release-wheels
# Creates: /tmp/release-wheels/ with wheels

# Show all options
./docker/run-docker-build.sh --help

# Test a PR from opm-simulators
./docker/run-docker-build.sh pr-test --version-simulators "pull/1234/merge" --output pr-wheels

# Test multiple PRs together
./docker/run-docker-build.sh multi-pr \
    --version-common "pull/567/merge" \
    --version-simulators "pull/1234/merge" \
    --output multi-pr-wheels

# Use specific branches/tags for each repository
./docker/run-docker-build.sh release \
    --version-common "release/2024.10/final" \
    --version-grid "release/2024.10/final" \
    --version-simulators "release/2024.10/final" \
    --output release-wheels

# Test PR #6075 with custom build targets
./docker/run-docker-build.sh pr6075 \
    --version-simulators "pull/6075/merge" \
    --target-simulators "GasWater BlackOil" \
    --output pr6075-wheels

Helper script features:

  • Works from any directory (auto-navigates to python/)
  • Smart output paths: Relative paths create directories in docker/, absolute paths used as-is
  • Automatic timestamped logging to docker/build-<tag>-<timestamp>.log
  • Reads Python versions from docker/python_versions.json by default
  • Input validation and helpful error messages
  • Progress indication and build summaries

Testing with Different Repository Versions

The build system supports testing different versions, branches, or pull requests from each OPM repository independently:

# Test a PR from opm-simulators against master branches
./docker/run-docker-build.sh pr-test \
    --version-simulators "pull/1234/merge" \
    --output pr-wheels

# Test compatibility between multiple PRs
./docker/run-docker-build.sh multi-pr \
    --version-common "pull/567/merge" \
    --version-grid "pull/890/merge" \
    --version-simulators "pull/1234/merge" \
    --output multi-pr-test

# Use specific release tags
./docker/run-docker-build.sh release \
    --version-common "release/2024.10/final" \
    --version-grid "release/2024.10/final" \
    --version-simulators "release/2024.10/final" \
    --output release-2024.10

The version parameters accept any valid Git reference:

  • Branch names: master, release/2024.10/final
  • Tags: v2024.10, release-2024-10
  • PR references: pull/1234/merge (GitHub PR format)
  • Commit hashes: abc123def456

The target parameters accept space-separated CMake target names:

  • Default targets: opmcommon_python (common), simulators (simulators)
  • PR #6075 targets: GasWater BlackOil (split simulators module)
  • Multiple targets: "target1 target2 target3"

Testing Wheels

See test_wheels/scripts/README.md for documentation on testing the generated wheels. The test framework installs the generated wheels in a clean Docker container and runs the complete test suites from both opm-common and opm-simulators to verify functionality.

Python Version Configuration

Python versions are configured in python_versions.json as the single source of truth.

Currently supported versions: Python 3.8, 3.9, 3.10, 3.11, 3.12, 3.13

Manylinux platform: manylinux_2_28_x86_64 (configured in docker/python_versions.json)

Adding a New Python Version

To add support for a new Python version (e.g., Python 3.14):

  1. Edit python_versions.json:

    "supported_versions": {
      "3.8": "3.8.19",
      "3.9": "3.9.21",
      "3.10": "3.10.16",
      "3.11": "3.11.11",
      "3.12": "3.12.9",
      "3.13": "3.13.1",
      "3.14": "3.14.0"  // Add new version with exact patch version
    }
    
  2. Run the sync script:

    ./docker/scripts/sync_versions.sh
    

    This automatically updates:

    • docker/Dockerfile (ARG python_versions default)
    • docker/test_wheels/scripts/README.md (supported versions list)
    • python/setup.py.in (python_requires minimum version)
    • Dynamic files read the JSON directly
  3. Manual update required: Edit docker/scripts/generate-pypi-package.sh lines 33-38 to add the new version mapping:

    all_python_versions[3.14]="cp314-cp314:/opt/python/cp314-cp314/bin/python"
    
  4. Test the new version:

    ./docker/run-docker-build.sh test --python-versions "3.14"
    

Removing/Deprecating a Python Version

To remove support for a Python version (e.g., Python 3.8):

  1. Edit python_versions.json: Remove the version from supported_versions
  2. Run the sync script: ./docker/scripts/sync_versions.sh
  3. Manual cleanup: Remove the version mapping from generate-pypi-package.sh
  4. Test: Ensure builds work without the removed version

Changing Manylinux Platform Version

To update the manylinux platform (e.g., from manylinux_2_28 to manylinux_2_34):

  1. Edit python_versions.json:

    "manylinux_platform": "manylinux_2_34_x86_64"
    
  2. Run the sync script:

    ./docker/scripts/sync_versions.sh
    

    This automatically updates:

    • docker/Dockerfile (FROM line with correct manylinux Docker image)
  3. Manual updates required:

    • docker/Dockerfile line 16: Update the compatibility comment
      • Example: Change "Debian 10+, Ubuntu 18.10+, CentOS/RHEL 8+" to match manylinux_2_34 requirements
    • docker/README.md line 166: Update the platform compatibility description
      • Verify and update the distribution compatibility information
  4. Test the new platform:

    ./docker/run-docker-build.sh test
    

Note: Each manylinux version has specific glibc requirements that determine distribution compatibility. Always verify the compatibility information when changing versions.

Maintenance Workflow

The JSON file controls version support across all build components:

Auto-synced locations (updated by sync script):

  • docker/Dockerfile - Default ARG python_versions
  • docker/test_wheels/scripts/README.md - Supported versions documentation
  • python/setup.py.in - Package python_requires minimum version
  • docker/test_wheels/scripts/src/opm_wheels/constants.py - Python version enum (dynamic)
  • docker/test_wheels/dockerfiles/common/install_python_versions.py - Installation script (dynamic)

Compatibility note (Windows checkout): directories under docker/test_wheels/dockerfiles use hyphens instead of colons. Example mappings: ubuntu:22.04 → ubuntu-22.04, ubuntu:24.04 → ubuntu-24.04, debian:11 → debian-11. The opm-wheels CLI accepts both forms via --docker-os, but uses the hyphenated form for filesystem paths and image tags.

Manual locations (requires manual update):

  • docker/scripts/generate-pypi-package.sh - Version to wheel tag mappings (lines 33-38)

Build Arguments

  • libtype: "static" (default) or "shared"
  • python_versions: Comma-separated list of Python versions (default: "3.8,3.9,3.10,3.11,3.12,3.13")
  • build_jobs: Number of parallel build jobs (default: 4)
  • version_common: Git reference for opm-common (default: "master")
  • version_grid: Git reference for opm-grid (default: "master")
  • version_simulators: Git reference for opm-simulators (default: "master")
  • target_common: CMake build targets for opm-common (default: "opmcommon_python")
  • target_simulators: CMake build targets for opm-simulators (default: "simulators")
  • version_tag: Version tag for wheels (optional)

Output

Built wheels follow the PEP 517 naming convention:

  • opm-{version}-{python}-{abi}-{platform}.whl
  • opm_simulators-{version}-{python}-{abi}-{platform}.whl

Platform tags use manylinux_2_28 for broad compatibility (Debian 10+, Ubuntu 18.10+, CentOS/RHEL 8+).