mirror of
https://github.com/OPM/opm-simulators.git
synced 2026-11-02 09:16:07 -06:00
Håkon Hægland
49494111d5
When a reservoir coupling slave fails during deck parsing, the master hangs on MPI_Recv waiting for initial data that never arrives. Fix by adding a status handshake after MPI_Comm_spawn: the slave sends OK/FAILED status before any data exchange, and the master checks it, disconnects, and throws on failure. Key changes: - Add SlaveStatus message tag and status handshake protocol - Slave: in readDeck(), override PARSE_MISSING_INCLUDE to throw instead of std::exit(), enabling cooperative MPI shutdown - Slave: send status to master in sendAndReceiveInitialData() - Master: receiveSlaveStatus_() checks status, disconnects all slave communicators and throws on failure - Main: notifyMasterSlaveInitFailed_() sends failure status from the readDeck() catch handler
89 lines
3.6 KiB
Bash
Executable File
89 lines
3.6 KiB
Bash
Executable File
#!/bin/bash
|
|
# Copyright 2026 Equinor
|
|
#
|
|
# This file is part of the Open Porous Media project (OPM).
|
|
#
|
|
# OPM is free software: you can redistribute it and/or modify
|
|
# it under the terms of the GNU General Public License as published by
|
|
# the Free Software Foundation, either version 3 of the License, or
|
|
# (at your option) any later version.
|
|
#
|
|
# OPM is distributed in the hope that it will be useful,
|
|
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
# GNU General Public License for more details.
|
|
#
|
|
# You should have received a copy of the GNU General Public License
|
|
# along with OPM. If not, see <http://www.gnu.org/licenses/>.
|
|
|
|
# CTest wrapper for the reservoir coupling slave parse error test.
|
|
#
|
|
# The slave deck has a deliberate parse error. The slave notifies the master
|
|
# of the failure via MPI, and the master exits with code 1 on all MPI
|
|
# implementations. Exit 0 (master completed despite slave parse error)
|
|
# is treated as FAIL.
|
|
#
|
|
# Known issue — OpenMPI 5.x BTL TCP IPv6 bug (observed with Ubuntu package):
|
|
# On systems where a network interface has both IPv4 and IPv6 addresses
|
|
# (common with WiFi), OpenMPI 5.x BTL TCP registers only the IPv6
|
|
# address for the interface when built with --enable-ipv6. When
|
|
# MPI_Comm_spawn is used, the spawned child may connect back to the
|
|
# parent using an IPv4 source address that the parent doesn't recognize,
|
|
# causing the connection to be rejected ("dropped inbound connection"
|
|
# or "UNREACHABLE").
|
|
#
|
|
# This bug is NOT present in OpenMPI 4.1.6. It was observed only with the
|
|
# Ubuntu 25.10 system package (5.0.8-8ubuntu1); custom source builds of the
|
|
# same version did not reproduce it, suggesting it depends on the specific
|
|
# OpenMPI build and runtime transport providers (e.g., --with-verbs,
|
|
# --with-libfabric). It could NOT be reproduced on the Ubuntu 26.04 system
|
|
# package (5.0.10-1): even with IPv6 registered first for the dual-stack
|
|
# interface (btl_tcp_disable_family unset), the spawned slave connects back
|
|
# successfully. The master-hang behavior this test detects is independent of
|
|
# this TCP bug — it stems from the slave exiting before the initial handshake.
|
|
#
|
|
# Root cause: opal/mca/btl/tcp/btl_tcp_component.c mca_btl_tcp_create()
|
|
# iterates opal_if_list and takes the FIRST address for each interface,
|
|
# which is IPv6 when both families are present. The source code has a
|
|
# comment acknowledging this limitation:
|
|
# https://github.com/open-mpi/ompi/blob/v5.0.8/opal/mca/btl/tcp/btl_tcp_component.c#L501-L515
|
|
#
|
|
# Workaround: disable IPv6 in BTL TCP address selection:
|
|
# export OMPI_MCA_btl_tcp_disable_family=6
|
|
# or persistently in ~/.openmpi/mca-params.conf:
|
|
# btl_tcp_disable_family = 6
|
|
# Note: the parameter uses literal 4/6, NOT AF_INET (2) / AF_INET6 (10).
|
|
#
|
|
# Usage: run_ctest.sh <flow_binary> <mpi_launcher>
|
|
|
|
set -u
|
|
|
|
FLOW_BINARY="${1:?Usage: $0 <flow_binary> <mpi_launcher>}"
|
|
MPI_LAUNCHER="${2:?Usage: $0 <flow_binary> <mpi_launcher>}"
|
|
|
|
echo "Flow binary: $FLOW_BINARY"
|
|
echo "MPI launcher: $MPI_LAUNCHER"
|
|
echo "Working dir: $(pwd)"
|
|
echo ""
|
|
|
|
# Run with a timeout as a safety net in case of regression (master hanging).
|
|
timeout 10 \
|
|
"$MPI_LAUNCHER" -np 1 "$FLOW_BINARY" \
|
|
RC_MASTER.DATA \
|
|
--parsing-strictness=low \
|
|
--output-dir=. \
|
|
2>&1
|
|
|
|
EXIT_CODE=$?
|
|
|
|
if [ $EXIT_CODE -eq 1 ]; then
|
|
echo "PASS: Master detected slave failure (exit code 1)"
|
|
exit 0
|
|
elif [ $EXIT_CODE -eq 0 ]; then
|
|
echo "FAIL: Master completed successfully — unexpected for slave parse error"
|
|
exit 1
|
|
else
|
|
echo "FAIL: Unexpected exit code $EXIT_CODE (expected 1)"
|
|
exit 1
|
|
fi
|