ResInsight serves gRPC requests from its main thread, so a long running call
leaves the heartbeat pings unanswered for as long as that call runs. The
heartbeat counted those timeouts as connection failures and closed the channel,
which aborted the very call the user was waiting for. The client reported
"RipsError: Channel closed!" while the ResInsight process was alive and well.
Classify a ping that fails with DEADLINE_EXCEEDED or RESOURCE_EXHAUSTED as a
busy server and never count it towards the failure threshold. Only failures
that indicate a broken connection declare the connection lost. A new
busy_warning_sec argument controls when a warning is logged for a server that
has been unresponsive for a while, so long operations are visible without being
killed.
Keep the process handle from Instance.launch so the heartbeat can check whether
ResInsight actually exited. A dead process is now detected immediately and
reported with its exit code instead of a bare channel error, and check_alive()
repeats that diagnosis.
Kristian Bendiksen
b0496f82d6