mirror of
https://github.com/OPM/opm-simulators.git
synced 2026-10-26 07:30:08 -05:00
Håkon Hægland
6e2503cac8
Reorganize Python folder: separate Docker/PyPI from CMake/PyBind11 workflows and implement comprehensive build automation This commit represents a comprehensive restructuring and enhancement of the OPM Python build system, addressing maintainability, consistency, and usability issues through architectural improvements and automation. FILES CHANGED ============= Refactored: - python/Dockerfile -> python/docker/Dockerfile - python/generate-pypi-package.sh -> python/docker/scripts/generate-pypi-package.sh - python/setup-docker-image.sh -> python/docker/scripts/setup-docker-image.sh - python/build-deps.sh -> python/docker/scripts/build-boost.sh - python/build-deps.sh -> python/docker/scripts/build-dune.sh Added (24 files): - python/docker/python_versions.json (single source of truth) - python/docker/scripts/sync_versions.sh - python/docker/scripts/read_python_config.py - python/docker/run-docker-build.sh - python/docker/README.md - python/docker/test_wheels/* (complete testing framework) Modified: - python/setup.py.in - .gitignore MAJOR CHANGES ============= Architecture Restructure ------------------------ - Separated build workflows: Clearly distinguished Docker/PyPI wheel generation from CMake/PyBind11 compilation - Moved Docker-specific files: Relocated docker build system to appropriate locations - Streamlined organization: Created logical separation between different build contexts Single Source of Truth Configuration ------------------------------------- - Centralized Python version management: Created python_versions.json as authoritative configuration - Automated synchronization: Built sync_versions.sh script to maintain consistency across all components - Dynamic configuration: Components now read from JSON rather than hardcoded values Enhanced Docker Build System ---------------------------- - Multi-OS support: Added Ubuntu 22.04, Ubuntu 24.04, and Debian 11 Dockerfiles - Improved caching: Split dependencies into separate Docker layers for better build performance - Fixed PyPI wheel generation: Added LD_LIBRARY_PATH setup for proper library bundling, corrected auditwheel wheel copying from wheelhouse/ instead of dist/ - Optimized Docker layer structure: Delayed source code copying until final stage, preventing dependency rebuilds on source changes and reducing iteration time from 30+ to <5 minutes during development Build Automation & Tooling -------------------------- - Comprehensive wheel testing: Created opm-wheels CLI tool for building and testing Docker images - Automated validation: Integrated Python version validation and error handling - Enhanced scripts: Improved generate-pypi-package.sh with better error handling and version management Testing Framework ----------------- - Docker-based testing: Complete framework for testing wheels across multiple Python versions and OS configurations - Automated validation: Wheel installation and import testing with comprehensive error reporting - Multi-version support: Parallel testing across Python 3.8-3.13 - Platform verification: Validates manylinux wheels install and function correctly across Ubuntu 22.04/24.04 and Debian 11 Documentation & Usability ------------------------- - Comprehensive maintenance guides: Detailed instructions for adding/removing Python versions - Clear workflows: Step-by-step procedures for build system maintenance - Improved CLI experience: Better error messages, path handling, and user guidance - Updated documentation: Enhanced README files with practical examples and troubleshooting IMPACT ====== This restructure transforms the Python build system from a collection of loosely coupled scripts into a cohesive, maintainable, and reliable build automation system. The single source of truth configuration eliminates the primary source of maintenance overhead, while the enhanced testing framework provides confidence in the quality of generated wheels. The changes enable the OPM project to more easily support new Python versions, maintain consistency across platforms, and provide a better developer experience for both maintainers and users.
72 lines
3.8 KiB
Docker
72 lines
3.8 KiB
Docker
# Dockerfile to generate PyPI packages. Needs to be run from the opm-simulators/python folder
|
|
#
|
|
# NOTE: Preferred way to build this docker image is to use docker/run-docker-build.sh
|
|
#
|
|
# Example use:
|
|
# docker build -t manylinux2014_opm:static . -f docker/Dockerfile
|
|
# docker build -t manylinux2014_opm:shared . -f docker/Dockerfile --build-arg libtype=shared
|
|
# docker build -t manylinux2014_opm:py312 . -f docker/Dockerfile --build-arg python_versions="3.12"
|
|
# docker build -t manylinux2014_opm:pr-test . -f docker/Dockerfile --build-arg version_simulators="pull/1234/head"
|
|
#
|
|
# To extract the wheel files from the final scratch image to a local directory, add --output:
|
|
# docker build -t manylinux2014_opm:static . -f docker/Dockerfile --output wheelhouse
|
|
# docker build -t manylinux2014_opm:shared . -f docker/Dockerfile --build-arg libtype=shared --output wheelhouse-shared
|
|
#
|
|
# Note: To see detailed build output instead of buildx progress, use:
|
|
# DOCKER_BUILDKIT=0 docker build -t manylinux2014_opm:built . -f docker/Dockerfile
|
|
# To redirect output to a log file for later inspection:
|
|
# docker build -t manylinux2014_opm:shared . -f docker/Dockerfile --build-arg libtype=shared 2>&1 | tee build.log
|
|
|
|
# Using manylinux_2_28 for broad compatibility (Debian 10+, Ubuntu 18.10+, CentOS/RHEL 8+)
|
|
# Newer manylinux_2_34 is available but has narrower distribution support
|
|
FROM quay.io/pypa/manylinux_2_28_x86_64 AS stage1
|
|
# Version control for each OPM repository (supports branches, tags, or PR refs like "pull/1234/head")
|
|
ARG version_common="master"
|
|
ARG version_grid="master"
|
|
ARG version_simulators="master"
|
|
ARG libtype="static" # "static" for static libraries (.a), "shared" for shared libraries (.so)
|
|
WORKDIR /tmp/opm
|
|
RUN if [ "$libtype" != "static" ] && [ "$libtype" != "shared" ]; then \
|
|
echo "Error: libtype must be 'static' or 'shared', got '$libtype'"; \
|
|
exit 1; \
|
|
fi
|
|
RUN echo "Using versions - Common: $version_common, Grid: $version_grid, Simulators: $version_simulators"
|
|
# Copy only the setup script to avoid invalidating cache with source code changes
|
|
COPY docker/scripts/setup-docker-image.sh /tmp/opm/setup-docker-image.sh
|
|
RUN /bin/bash setup-docker-image.sh $libtype
|
|
|
|
# Multi-stage build: stage1=setup, stage2=build deps, stage3=generate packages, export_stage=extract results
|
|
FROM stage1 AS stage2
|
|
ARG libtype="static"
|
|
ARG build_jobs=4
|
|
# Build dependencies in separate layers for better Docker caching
|
|
# Copy only dependency build scripts (delay "ADD . .") to preserve cache layers for better debugging
|
|
# experience. I.e. if you change the source code, the build will not be invalidated and you have to
|
|
# rebuild everything from scratch.
|
|
COPY docker/scripts/build-boost.sh /tmp/opm/build-boost.sh
|
|
RUN /bin/bash build-boost.sh $build_jobs $libtype
|
|
COPY docker/scripts/build-dune.sh /tmp/opm/build-dune.sh
|
|
RUN /bin/bash build-dune.sh $build_jobs $libtype
|
|
|
|
FROM stage2 AS stage3
|
|
# Version control for each OPM repository
|
|
ARG version_common="master"
|
|
ARG version_grid="master"
|
|
ARG version_simulators="master"
|
|
ARG libtype="static"
|
|
ARG build_jobs=4
|
|
ARG version_tag=""
|
|
# CMake build targets (default for master branch, may differ in PRs)
|
|
ARG target_common="opmcommon_python"
|
|
ARG target_simulators="simulators"
|
|
# Default Python versions - can be overridden via build-arg or read from docker/python_versions.json
|
|
# NOTE: The list is maintained in docker/python_versions.json and docker/scripts/sync_versions.sh
|
|
ARG python_versions="3.8,3.9,3.10,3.11,3.12,3.13"
|
|
# Finally copy all source files - only needed for wheel generation
|
|
ADD . .
|
|
RUN /bin/bash docker/scripts/generate-pypi-package.sh "$version_common" "$version_grid" "$version_simulators" "$build_jobs" "$version_tag" $libtype "$python_versions" "$target_common" "$target_simulators"
|
|
|
|
# Final stage: extract only the wheel files using minimal scratch image
|
|
FROM scratch AS export_stage
|
|
COPY --from=stage3 /tmp/opm/wheelhouse .
|