mirror of
https://github.com/OPM/opm-simulators.git
synced 2026-11-04 21:27:12 -06:00
Håkon Hægland
6e2503cac8
Reorganize Python folder: separate Docker/PyPI from CMake/PyBind11 workflows and implement comprehensive build automation This commit represents a comprehensive restructuring and enhancement of the OPM Python build system, addressing maintainability, consistency, and usability issues through architectural improvements and automation. FILES CHANGED ============= Refactored: - python/Dockerfile -> python/docker/Dockerfile - python/generate-pypi-package.sh -> python/docker/scripts/generate-pypi-package.sh - python/setup-docker-image.sh -> python/docker/scripts/setup-docker-image.sh - python/build-deps.sh -> python/docker/scripts/build-boost.sh - python/build-deps.sh -> python/docker/scripts/build-dune.sh Added (24 files): - python/docker/python_versions.json (single source of truth) - python/docker/scripts/sync_versions.sh - python/docker/scripts/read_python_config.py - python/docker/run-docker-build.sh - python/docker/README.md - python/docker/test_wheels/* (complete testing framework) Modified: - python/setup.py.in - .gitignore MAJOR CHANGES ============= Architecture Restructure ------------------------ - Separated build workflows: Clearly distinguished Docker/PyPI wheel generation from CMake/PyBind11 compilation - Moved Docker-specific files: Relocated docker build system to appropriate locations - Streamlined organization: Created logical separation between different build contexts Single Source of Truth Configuration ------------------------------------- - Centralized Python version management: Created python_versions.json as authoritative configuration - Automated synchronization: Built sync_versions.sh script to maintain consistency across all components - Dynamic configuration: Components now read from JSON rather than hardcoded values Enhanced Docker Build System ---------------------------- - Multi-OS support: Added Ubuntu 22.04, Ubuntu 24.04, and Debian 11 Dockerfiles - Improved caching: Split dependencies into separate Docker layers for better build performance - Fixed PyPI wheel generation: Added LD_LIBRARY_PATH setup for proper library bundling, corrected auditwheel wheel copying from wheelhouse/ instead of dist/ - Optimized Docker layer structure: Delayed source code copying until final stage, preventing dependency rebuilds on source changes and reducing iteration time from 30+ to <5 minutes during development Build Automation & Tooling -------------------------- - Comprehensive wheel testing: Created opm-wheels CLI tool for building and testing Docker images - Automated validation: Integrated Python version validation and error handling - Enhanced scripts: Improved generate-pypi-package.sh with better error handling and version management Testing Framework ----------------- - Docker-based testing: Complete framework for testing wheels across multiple Python versions and OS configurations - Automated validation: Wheel installation and import testing with comprehensive error reporting - Multi-version support: Parallel testing across Python 3.8-3.13 - Platform verification: Validates manylinux wheels install and function correctly across Ubuntu 22.04/24.04 and Debian 11 Documentation & Usability ------------------------- - Comprehensive maintenance guides: Detailed instructions for adding/removing Python versions - Clear workflows: Step-by-step procedures for build system maintenance - Improved CLI experience: Better error messages, path handling, and user guidance - Updated documentation: Enhanced README files with practical examples and troubleshooting IMPACT ====== This restructure transforms the Python build system from a collection of loosely coupled scripts into a cohesive, maintainable, and reliable build automation system. The single source of truth configuration eliminates the primary source of maintenance overhead, while the enhanced testing framework provides confidence in the quality of generated wheels. The changes enable the OPM project to more easily support new Python versions, maintain consistency across platforms, and provide a better developer experience for both maintainers and users.
Python scripts for testing the generated wheels
Installation
For regular use:
$ python -m venv .venv
$ source .venv/bin/activate
$ pip install . # Installs package and dependencies
For development (editable install with locked dependency versions):
$ uv venv
$ uv sync # Installs package in editable mode with exact versions from uv.lock
$ source .venv/bin/activate
Build the PyPI wheels
Build the wheels from the python folder of the opm-simulators repository using
the provided helper script for simplified building with automatic logging:
# Static build with wheel extraction
./docker/run-docker-build.sh static --output wheelhouse
# Creates: docker/wheelhouse/ with wheels
# Shared libraries build
./docker/run-docker-build.sh shared --libtype shared --output wheelhouse-shared
# Creates: docker/wheelhouse-shared/ with wheels
see .docker/run-docker-build.sh for more examples.
Build a test container
Build a Docker container for testing the generated wheels:
# Basic usage (Ubuntu 24.04)
$ opm-wheels build-docker-image --docker-os="ubuntu:24.04"
# Build with custom repository and branch
$ opm-wheels build-docker-image \
--docker-os="ubuntu:22.04" \
--opm-simulators-branch="pr-branch" \
--opm-common-branch="development" \
--docker-progress
# Build for Debian
$ opm-wheels build-docker-image --docker-os="debian:11"
Available options for build-docker-image:
--docker-os,-d- OS for Docker image (default: ubuntu:22.04)- Supported:
3.8,3.9,3.10,3.11,3.12,3.13
- Supported:
--opm-simulators-repo,-osr- OPM simulators repository URL--opm-common-repo,-ocr- OPM common repository URL--opm-simulators-branch,-osb- Simulators branch (default: master)--opm-common-branch,-ocb- Common branch (default: master)--docker-progress,-dp- Show Docker build progress output
Run tests
Run unit tests for opm-common and opm-simulators wheels in the test container:
# Test all Python versions with default wheel directory
$ opm-wheels run-docker-image --docker-os="ubuntu:24.04"
# Test specific Python versions
$ opm-wheels run-docker-image \
--docker-os="ubuntu:22.04" \
--python-versions="3.11,3.12,3.13"
# Test with custom wheel directory
$ opm-wheels run-docker-image \
--docker-os="debian:11" \
--wheel-dir="/path/to/wheelhouse-shared" \
--python-versions="3.12"
Available options for run-docker-image:
--docker-os,-d- OS for Docker image (default: ubuntu:22.04)--wheel-dir- Directory containing wheel files (default: python/wheelhouse)--python-versions,-p- Python versions for testing- Supported:
3.8,3.9,3.10,3.11,3.12,3.13 - Format: comma-separated list without spaces
- Supported:
Additional commands
Show wheel contents
# List files inside a wheel
$ opm-wheels show-wheel-files opm-2025.10-cp312-cp312-manylinux_2_27_x86_64.whl
# Use custom wheel directory
$ opm-wheels show-wheel-files \
--wheel-dir="wheelhouse-shared" \
opm_simulators-2025.10-cp312-cp312-manylinux_2_27_x86_64.whl