mirror of
https://github.com/OPM/opm-simulators.git
synced 2026-10-24 22:57:42 -05:00
Håkon Hægland
6e2503cac8
Reorganize Python folder: separate Docker/PyPI from CMake/PyBind11 workflows and implement comprehensive build automation This commit represents a comprehensive restructuring and enhancement of the OPM Python build system, addressing maintainability, consistency, and usability issues through architectural improvements and automation. FILES CHANGED ============= Refactored: - python/Dockerfile -> python/docker/Dockerfile - python/generate-pypi-package.sh -> python/docker/scripts/generate-pypi-package.sh - python/setup-docker-image.sh -> python/docker/scripts/setup-docker-image.sh - python/build-deps.sh -> python/docker/scripts/build-boost.sh - python/build-deps.sh -> python/docker/scripts/build-dune.sh Added (24 files): - python/docker/python_versions.json (single source of truth) - python/docker/scripts/sync_versions.sh - python/docker/scripts/read_python_config.py - python/docker/run-docker-build.sh - python/docker/README.md - python/docker/test_wheels/* (complete testing framework) Modified: - python/setup.py.in - .gitignore MAJOR CHANGES ============= Architecture Restructure ------------------------ - Separated build workflows: Clearly distinguished Docker/PyPI wheel generation from CMake/PyBind11 compilation - Moved Docker-specific files: Relocated docker build system to appropriate locations - Streamlined organization: Created logical separation between different build contexts Single Source of Truth Configuration ------------------------------------- - Centralized Python version management: Created python_versions.json as authoritative configuration - Automated synchronization: Built sync_versions.sh script to maintain consistency across all components - Dynamic configuration: Components now read from JSON rather than hardcoded values Enhanced Docker Build System ---------------------------- - Multi-OS support: Added Ubuntu 22.04, Ubuntu 24.04, and Debian 11 Dockerfiles - Improved caching: Split dependencies into separate Docker layers for better build performance - Fixed PyPI wheel generation: Added LD_LIBRARY_PATH setup for proper library bundling, corrected auditwheel wheel copying from wheelhouse/ instead of dist/ - Optimized Docker layer structure: Delayed source code copying until final stage, preventing dependency rebuilds on source changes and reducing iteration time from 30+ to <5 minutes during development Build Automation & Tooling -------------------------- - Comprehensive wheel testing: Created opm-wheels CLI tool for building and testing Docker images - Automated validation: Integrated Python version validation and error handling - Enhanced scripts: Improved generate-pypi-package.sh with better error handling and version management Testing Framework ----------------- - Docker-based testing: Complete framework for testing wheels across multiple Python versions and OS configurations - Automated validation: Wheel installation and import testing with comprehensive error reporting - Multi-version support: Parallel testing across Python 3.8-3.13 - Platform verification: Validates manylinux wheels install and function correctly across Ubuntu 22.04/24.04 and Debian 11 Documentation & Usability ------------------------- - Comprehensive maintenance guides: Detailed instructions for adding/removing Python versions - Clear workflows: Step-by-step procedures for build system maintenance - Improved CLI experience: Better error messages, path handling, and user guidance - Updated documentation: Enhanced README files with practical examples and troubleshooting IMPACT ====== This restructure transforms the Python build system from a collection of loosely coupled scripts into a cohesive, maintainable, and reliable build automation system. The single source of truth configuration eliminates the primary source of maintenance overhead, while the enhanced testing framework provides confidence in the quality of generated wheels. The changes enable the OPM project to more easily support new Python versions, maintain consistency across platforms, and provide a better developer experience for both maintainers and users.
35 lines
1.8 KiB
JSON
35 lines
1.8 KiB
JSON
{
|
|
"_comment": "SINGLE SOURCE OF TRUTH for Python versions in OPM wheel build system",
|
|
"_maintenance": {
|
|
"description": "This file controls Python version support across all Docker build components",
|
|
"after_editing": [
|
|
"1. Run: ./docker/scripts/sync_versions.sh",
|
|
"2. Manually update docker/scripts/generate-pypi-package.sh lines 33-38 (version mappings)",
|
|
"3. Test build: ./docker/run-docker-build.sh test --python-versions [your versions]"
|
|
],
|
|
"locations_auto_synced": [
|
|
"docker/Dockerfile (ARG python_versions default - derived from supported_versions)",
|
|
"docker/test_wheels/scripts/README.md (supported versions list - derived from supported_versions)",
|
|
"docker/test_wheels/scripts/src/opm_wheels/constants.py (enum auto-generated from supported_versions)",
|
|
"docker/test_wheels/dockerfiles/common/install_python_versions.py (reads supported_versions directly)",
|
|
"python/setup.py.in (python_requires minimum version - derived from supported_versions)"
|
|
],
|
|
"locations_manual": [
|
|
"docker/scripts/generate-pypi-package.sh (version mappings array lines 41-46)",
|
|
"docker/Dockerfile line 16 (manylinux compatibility comment - verify distribution compatibility)",
|
|
"docker/README.md line 166 (platform compatibility description - verify distribution compatibility)"
|
|
],
|
|
"note": "Default versions are automatically derived from supported_versions.keys() - no separate default_versions field needed"
|
|
},
|
|
"supported_versions": {
|
|
"3.8": "3.8.19",
|
|
"3.9": "3.9.21",
|
|
"3.10": "3.10.16",
|
|
"3.11": "3.11.11",
|
|
"3.12": "3.12.9",
|
|
"3.13": "3.13.1"
|
|
},
|
|
"manylinux_platform": "manylinux_2_28_x86_64",
|
|
"_manylinux_note": "auditwheel may optimize to an older version (e.g., manylinux_2_27) for broader compatibility"
|
|
}
|