opm-simulators/python/docker/README.md

# Docker Build System for OPM Python Wheels

This directory contains the Docker-based build system for generating PyPI-compatible wheels for the OPM Python bindings.

## Directory Structure

```
docker/
├── Dockerfile                 # Main Docker build configuration
├── run-docker-build.sh        # User-facing helper script for building wheels
├── python_versions.json       # Python version definitions
├── README.md                  # This file (documentation)
├── scripts/                   # Internal build scripts
│   ├── generate-pypi-package.sh    # Wheel generation script
│   ├── setup-docker-image.sh       # Docker environment setup
│   ├── build-boost.sh             # Boost dependency build
│   ├── build-dune.sh              # DUNE dependency build
│   ├── build-deps.sh              # Additional dependencies
│   └── read_python_config.py      # Python version config reader
├── test_wheels/               # Wheel testing framework
└── [output directories]       # Generated by --output parameter (user-specified names)
```

## Building Wheels

Use the provided helper script for simplified building with automatic logging:

```bash
# Static build with wheel extraction
./docker/run-docker-build.sh static --output wheelhouse
# Creates: docker/wheelhouse/ with wheels

# Shared libraries build
./docker/run-docker-build.sh shared --libtype shared --output wheelhouse-shared
# Creates: docker/wheelhouse-shared/ with wheels

# Build specific Python versions
./docker/run-docker-build.sh py312 --python-versions "3.12" --output test-wheels
# Creates: docker/test-wheels/ with wheels

# Debug build with detailed output (no wheel extraction)
./docker/run-docker-build.sh debug --libtype shared --no-buildx --jobs 8

# Absolute path output (advanced usage)
./docker/run-docker-build.sh release --output /tmp/release-wheels
# Creates: /tmp/release-wheels/ with wheels

# Show all options
./docker/run-docker-build.sh --help

# Test a PR from opm-simulators
./docker/run-docker-build.sh pr-test --version-simulators "pull/1234/merge" --output pr-wheels

# Test multiple PRs together
./docker/run-docker-build.sh multi-pr \
    --version-common "pull/567/merge" \
    --version-simulators "pull/1234/merge" \
    --output multi-pr-wheels

# Use specific branches/tags for each repository
./docker/run-docker-build.sh release \
    --version-common "release/2024.10/final" \
    --version-grid "release/2024.10/final" \
    --version-simulators "release/2024.10/final" \
    --output release-wheels

# Test PR #6075 with custom build targets
./docker/run-docker-build.sh pr6075 \
    --version-simulators "pull/6075/merge" \
    --target-simulators "GasWater BlackOil" \
    --output pr6075-wheels
```

**Helper script features:**
- Works from any directory (auto-navigates to `python/`)
- **Smart output paths**: Relative paths create directories in `docker/`, absolute paths used as-is
- Automatic timestamped logging to `docker/build-<tag>-<timestamp>.log`
- Reads Python versions from `docker/python_versions.json` by default
- Input validation and helpful error messages
- Progress indication and build summaries

## Testing with Different Repository Versions

The build system supports testing different versions, branches, or pull requests from each OPM repository independently:

```bash
# Test a PR from opm-simulators against master branches
./docker/run-docker-build.sh pr-test \
    --version-simulators "pull/1234/merge" \
    --output pr-wheels

# Test compatibility between multiple PRs
./docker/run-docker-build.sh multi-pr \
    --version-common "pull/567/merge" \
    --version-grid "pull/890/merge" \
    --version-simulators "pull/1234/merge" \
    --output multi-pr-test

# Use specific release tags
./docker/run-docker-build.sh release \
    --version-common "release/2024.10/final" \
    --version-grid "release/2024.10/final" \
    --version-simulators "release/2024.10/final" \
    --output release-2024.10
```

The version parameters accept any valid Git reference:
- Branch names: `master`, `release/2024.10/final`
- Tags: `v2024.10`, `release-2024-10`
- PR references: `pull/1234/merge` (GitHub PR format)
- Commit hashes: `abc123def456`

The target parameters accept space-separated CMake target names:
- Default targets: `opmcommon_python` (common), `simulators` (simulators)
- PR #6075 targets: `GasWater BlackOil` (split simulators module)
- Multiple targets: `"target1 target2 target3"`

## Testing Wheels

See [test_wheels/scripts/README.md](test_wheels/scripts/README.md) for documentation on testing the generated
wheels. The test framework installs the generated wheels in a clean Docker container and runs the complete test suites from both opm-common and opm-simulators to verify functionality.

## Python Version Configuration

Python versions are configured in [`python_versions.json`](python_versions.json) as the **single source of truth**.

**Currently supported versions:** Python 3.8, 3.9, 3.10, 3.11, 3.12, 3.13

**Manylinux platform:** manylinux_2_28_x86_64 (configured in `docker/python_versions.json`)

### Adding a New Python Version

To add support for a new Python version (e.g., Python 3.14):

1. **Edit `python_versions.json`:**
   ```json
   "supported_versions": {
     "3.8": "3.8.19",
     "3.9": "3.9.21",
     "3.10": "3.10.16",
     "3.11": "3.11.11",
     "3.12": "3.12.9",
     "3.13": "3.13.1",
     "3.14": "3.14.0"  // Add new version with exact patch version
   }
   ```

2. **Run the sync script:**
   ```bash
   ./docker/scripts/sync_versions.sh
   ```
   This automatically updates:
   - `docker/Dockerfile` (ARG python_versions default)
   - `docker/test_wheels/scripts/README.md` (supported versions list)
   - `python/setup.py.in` (python_requires minimum version)
   - Dynamic files read the JSON directly

3. **Manual update required:**
   Edit `docker/scripts/generate-pypi-package.sh` lines 33-38 to add the new version mapping:
   ```bash
   all_python_versions[3.14]="cp314-cp314:/opt/python/cp314-cp314/bin/python"
   ```

4. **Test the new version:**
   ```bash
   ./docker/run-docker-build.sh test --python-versions "3.14"
   ```

### Removing/Deprecating a Python Version

To remove support for a Python version (e.g., Python 3.8):

1. **Edit `python_versions.json`:** Remove the version from `supported_versions`
2. **Run the sync script:** `./docker/scripts/sync_versions.sh`
3. **Manual cleanup:** Remove the version mapping from `generate-pypi-package.sh`
4. **Test:** Ensure builds work without the removed version

### Changing Manylinux Platform Version

To update the manylinux platform (e.g., from manylinux_2_28 to manylinux_2_34):

1. **Edit `python_versions.json`:**
   ```json
   "manylinux_platform": "manylinux_2_34_x86_64"
   ```

2. **Run the sync script:**
   ```bash
   ./docker/scripts/sync_versions.sh
   ```
   This automatically updates:
   - `docker/Dockerfile` (FROM line with correct manylinux Docker image)

3. **Manual updates required:**
   - **docker/Dockerfile line 16**: Update the compatibility comment
     - Example: Change "Debian 10+, Ubuntu 18.10+, CentOS/RHEL 8+" to match manylinux_2_34 requirements
   - **docker/README.md line 166**: Update the platform compatibility description
     - Verify and update the distribution compatibility information

4. **Test the new platform:**
   ```bash
   ./docker/run-docker-build.sh test
   ```

**Note:** Each manylinux version has specific glibc requirements that determine distribution compatibility. Always verify the compatibility information when changing versions.

### Maintenance Workflow

The JSON file controls version support across all build components:

**Auto-synced locations** (updated by sync script):
- `docker/Dockerfile` - Default ARG python_versions
- `docker/test_wheels/scripts/README.md` - Supported versions documentation
- `python/setup.py.in` - Package python_requires minimum version
- `docker/test_wheels/scripts/src/opm_wheels/constants.py` - Python version enum (dynamic)
- `docker/test_wheels/dockerfiles/common/install_python_versions.py` - Installation script (dynamic)

Compatibility note (Windows checkout): directories under `docker/test_wheels/dockerfiles`
use hyphens instead of colons. Example mappings: `ubuntu:22.04` → `ubuntu-22.04`,
`ubuntu:24.04` → `ubuntu-24.04`, `debian:11` → `debian-11`. The `opm-wheels` CLI
accepts both forms via `--docker-os`, but uses the hyphenated form for filesystem
paths and image tags.

**Manual locations** (requires manual update):
- `docker/scripts/generate-pypi-package.sh` - Version to wheel tag mappings (lines 33-38)

## Build Arguments

- `libtype`: "static" (default) or "shared"
- `python_versions`: Comma-separated list of Python versions (default: "3.8,3.9,3.10,3.11,3.12,3.13")
- `build_jobs`: Number of parallel build jobs (default: 4)
- `version_common`: Git reference for opm-common (default: "master")
- `version_grid`: Git reference for opm-grid (default: "master")
- `version_simulators`: Git reference for opm-simulators (default: "master")
- `target_common`: CMake build targets for opm-common (default: "opmcommon_python")
- `target_simulators`: CMake build targets for opm-simulators (default: "simulators")
- `version_tag`: Version tag for wheels (optional)

## Output

Built wheels follow the PEP 517 naming convention:
- `opm-{version}-{python}-{abi}-{platform}.whl`
- `opm_simulators-{version}-{python}-{abi}-{platform}.whl`

Platform tags use manylinux_2_28 for broad compatibility (Debian 10+, Ubuntu 18.10+, CentOS/RHEL 8+).