# Workflow UI design ## What this branch adds A new **Workflows** node in the project tree, next to **Scripts**. It lists [taskmaestro](https://github.com/OPM/taskmaestro) workflows found on disk and the workflows that installed packages register. For each one it auto-generates a property editor for the config inputs, and it runs the workflow as a subprocess that talks back to ResInsight over the existing gRPC channel. Workflows can also be **created and edited** in ResInsight. A graph editor with a task palette builds the workflow from registered tasks and saves it as a standard `workflow.yaml` + `input.yaml`. The task classes are the source of truth. ResInsight reads the Pydantic input and output models of each task, which are exported as JSON schemas, to build the property editors and to check which ports can be connected. There is no manifest file or plugin descriptor. ## Architecture at a glance ``` ResInsight (C++) Python helper (rips.taskmaestro_helper) ───────────────── ─────────────────────────────────────── RimWorkflowTaskCatalog ────── catalog ─────► entry points → task descriptors + registered workflows RimWorkflowCollection folders in ~/.taskmaestro/workflows ─ load ─► workflow.yaml + input.yaml → definition JSON Installed (RimWorkflowInstalledCollection) (from the catalog) RimWorkflow RimWorkflowDefinition (value struct) edits → applyEdit → undo/redo, dirty Save ──────────────── save (stdin JSON) ──► definition JSON → workflow.yaml + input.yaml RiuWorkflowValidationRunner ─ save --describe ► temp folder + taskmaestro describe RimWorkflowJob (one or more) RimWorkflowTaskInput (one per task) RimWorkflowBinding (one per config field, see table below) Run ─ writes input.yaml ─ QProcess ─────► run | run --registered ◄───── JSON events → task states in the graph ``` The C++ side never parses YAML and doesn't import Pydantic. It only handles JSON (Qt's `QJsonDocument`); `input.yaml` for a run is written as JSON text, which YAML accepts. The Python helper does everything that touches YAML or the upstream library. Every helper call goes through `RimWorkflowHelperProcess::runSync` (or the async validation runner), which finds the Python interpreter and decodes the single JSON document on stdout. ## The Python helper `python -m rips.taskmaestro_helper ` ([__main__.py](../GrpcInterface/Python/rips/taskmaestro_helper/__main__.py)). Every subcommand except `run` prints exactly one JSON document. Output written while task modules are imported goes to stderr. On failure the helper exits with code 2 and prints `{"status": "invalid", "error": {"message", "task", "field"}}`. taskmaestro 0.4.0 or newer (below 0.5) is required (the `workflow` extra of `rips`); a missing or older taskmaestro is reported as an import error. | Subcommand | Purpose | |---|---| | `catalog [--no-workflows]` | All tasks from the `taskmaestro.tasks` entry points, as descriptors (`id`, `name`, `python_type`, `description`, `input_schema`, `output_schema`), and all workflows from `taskmaestro.workflows`. A broken plugin is reported in `errors`; the rest still load. | | `load [--describe]` | Reads `workflow.yaml` + `input.yaml` into a definition JSON (below). | | `save --out-dir D [--workflow-only] [--describe] [--backup]` | Reads a definition JSON on stdin and writes the YAML files atomically. `--describe` also runs `taskmaestro workflow describe` on the result, which is how validation works. | | `export-registered [--describe]` | A registered workflow as a definition JSON. | | `run ` / `run --registered ` | Runs a workflow (see [The run flow](#the-run-flow)). | `schema.py` generates the task schemas. It adds `x-ri-python-type` and `x-ri-python-bases` (the class MRO) to models and opaque types, so that C++ can check subclass relations without Python. ## The definition model `RimWorkflowDefinition` ([RimWorkflowDefinition.h](../ApplicationLibCode/ProjectDataModel/Workflow/RimWorkflowDefinition.h)) is a plain value struct that mirrors the helper's definition JSON: ```json { "format": 1, "name": "resinsight_completions", "result_task": null, "editable": true, "readonly_reasons": [], "warnings": [], "header_comment": "# ...", "passthrough": {"runner": {}}, "nodes": [{"name": "add_perf_1", "task": "resinsight.add_perforation", "config_fields": ["event_date", "start_md", "end_md"]}], "edges": [{"from": "connect_to_resinsight", "output": null, "to": "add_perf_1", "input": "resinsight"}], "inputs": {"add_perf_1": {"start_md": 3000.0}}, "task_types": {"resinsight.add_perforation": {"...": "descriptor"}}, "describe": {}, "describe_error": null } ``` - A `null` `output`/`input` on an edge means the whole model. `[task, field]` routing and fan-in (`{field: task}`) are both expressed as edges. - `task_types` embeds the descriptor of every task used, so a workflow can still be edited when the catalog fails to load. - `save` always writes `name:` on every entry and refers to tasks by instance name, because taskmaestro cannot resolve default names in `depends_on`. Linear (implicit) workflows are converted to explicit edges on load. - The definition is never written to the `.rsp` file. Undo and redo store value copies (`RimWorkflow`, capped at 100 steps). All edits are pure functions in `RimWorkflowDefinitionTools` that return `std::expected`: `addTask`, `removeTask`, `connect`, `disconnect`, `renameTask`, `renameWorkflow`, `setResultTask` and `setOptionalInputConfigured`. After each edit, `normalize` re-derives the config fields: every required input field that no edge covers becomes a config field, as do optional fields the user has turned on. `addTask` gives the node a unique instance name. It also auto-wires inputs of type `RipsInstance` to a `resinsight.connect` node, which is created the first time it is needed. ### Read-only workflows A workflow opens read-only, showing the reasons in the editor's status line, when: - it is **installed** (registered by a package). Use *Duplicate as Editable* to get an editable copy. - it uses features the editor does not support: nested `workflow:` entries, or task classes that cannot be imported. Read-only workflows can still be run. ## Port typing `RimWorkflowPortCompatibility` lists the input and output ports of a task from its schemas. The whole model is a port with an empty name, and the opaque `value` field of an `ObjectModel` is hidden. `isCompatible(produced, expected)` checks the following, in order: 1. `Any` accepts everything. 2. `anyOf`: every produced option must match; at least one expected option must match. 3. Opaque types: the expected Python type must be among the produced type's bases, so `PerforationOutput` does not connect to `WellPath` even though both wrap `rips.WellPath`. 4. Models: compare `x-ri-python-type` against the produced bases. 5. Arrays: compare the items. 6. Scalars: `type` and `format` must be equal. bool → int is allowed; int → float is not. Enums must be a subset. `canConnect` also rejects self-loops, cycles, and a whole-input edge on a task that already has field edges (and the reverse). The error text is shown as a tooltip while dragging. ## Workflow graph and editor Selecting a `RimWorkflow` (or one of its jobs) shows `RiuWorkflowEditorWidget` in a central dock tab: - the graph (`RiuWorkflowGraphView`); - the task palette (`RiuWorkflowTaskPalette`) to the right of the graph: a filter field above a tree grouped by id prefix. Drag a task onto the graph, or double-click it; - a status line for load errors, read-only reasons, validation results and unsaved changes. Each port shows its name with its type below it in smaller text (from the task schemas, as `input_types`/`output_types` in the graph JSON). The tooltip also shows the description. Ports that hold a ResInsight object (instance, case, view, well path, surface) show that object's project tree icon in front of the type (`input_icons`/`output_icons`). The rips class is found in the `x-ri-python-bases`, either directly or as `ObjectModel[Class]` (`RimWorkflowPortCompatibility::iconResource`). Other ports show an icon for their kind of value: string, integer, float, bool, date, file path or directory. These are Codicons (CC BY 4.0), kept with their licence and source in `ApplicationExeCode/Resources/codicons`. `RiuWorkflowGraphLayout` computes the layout: topological ranks with a Kahn sort and longest-path ranks, then one barycenter pass to order nodes. It tolerates cycles. The layout is always automatic; nodes moved by hand keep their position only for the session. After an edit the viewport is kept. In an editable workflow: - Drag from an output port to an input port to connect. Valid targets turn green, invalid ones grey with the reason as a tooltip. Escape cancels. - Unwired optional inputs are drawn as hollow ports. - Right-click a task for Rename (F2), Set as Result Task, Optional Inputs ▸ and Delete (Del). Right-click a connection for Delete Connection. Right-click the background for Add Task ▸, Undo, Redo, Save, Save As, Duplicate as Editable and Fit. There is no toolbar. - Ctrl+Z/Ctrl+Y undo and redo; Ctrl+S saves. ### Collect and map `collect` (taskmaestro 0.3) gathers several outputs into one input. In the definition each member is a separate edge with `collect: list|dict` (and a `key` for dicts); the order of the list edges is the list order. `RimWorkflowPortCompatibility::resolveConnection` decides whether a drop is a plain edge or a collect member: an input typed `list[T]` or `dict[str, T]` collects outputs of type `T`, while an output of the whole container type still connects directly. Collecting inputs are drawn with a ring around the port; member edges are labelled `[i]` or with their key. Dict keys default to the source task name and are made unique. Right-click a member for Rename Key… (dict) or Move Earlier/Later (list). `map` runs a task once per item of a configured mapping. Right-click a task for Map Over… (Edit Map… when mapped) and Remove Map. The dialog picks the mapping's config name (`over`), the inputs that receive the key (`key_as`, must accept a string) and the value (`value_as`), and the error mode (`fail_fast` or `collect_all`); `checkTaskMap` validates it while typing. `taskTypeForNode` gives the effective task type of a mapped node: `key_as` and `value_as` are no longer inputs, and the output becomes `dict[str, Output]`. `graphTask` adds `over` as a config field and port (`map_over: true`) typed `dict[str, ]` (`mapOverSchema`). The node is drawn as a stack with a "⟳ map over X" subtitle. The `over` value is edited with `RimWorkflowMappingBinding`: a JSON object, or a multi-selection of cases, well paths or views keyed by name. During a run the helper emits `map_item_state` events; the node shows "done/seen" counts and failures, and the tooltip lists each item. Selecting a task shows its `RimWorkflowTaskInput` from the displayed job in the property view. Editing is locked while a job of the workflow is running. During a run, task states are drawn on the nodes. ## Validation Two layers of validation run: - **Static**, in C++ (`RimWorkflowDefinitionTools::validate`): unknown tasks, invalid names, required opaque inputs that are not wired, multiple sinks without a result task, and so on. The results are shown as badges on the nodes right away. - **Live**, through taskmaestro: `RiuWorkflowValidationRunner` waits 400 ms after the last edit, then runs `save --out-dir --describe` in the background. A new edit kills a stale run, results for an old revision are dropped, and a 30 s watchdog stops hung processes. `RimWorkflowValidationTools::issuesFromHelperResult` maps the errors to tasks; anything left over goes to the status line. - **Missing values** come from the displayed job, not from describe. The saved definition only carries literal values, so describe would report cases, well paths and views as missing. The editor replaces describe's "has no value" warnings with ones computed from the job's bindings (`RimWorkflowValidationTools::issuesWithJobValues`). They update as soon as a value is edited. Running a workflow whose last validation failed asks for confirmation first. ## Saving - **Save** writes `workflow.yaml` and `input.yaml` in the workflow folder. The first save makes `workflow.yaml.bak`. Before overwriting, ResInsight warns if the file changed on disk since it was loaded, or if the folder is a symlink into a git checkout. - **Save As** and **New Workflow** ask for a name and a folder. The default folder is `~/.taskmaestro/workflows/`; a folder elsewhere is not discovered on the next rescan, and ResInsight warns about that. `*.py` files next to the original `workflow.yaml` are copied along, so tasks given as a dotted path (`pipeline.MyTask`) still import. - `input.yaml` gets the literal values of the **first job** (the "Default" job). Object references (cases, well paths, views) are not saved. - YAML comments other than the leading header block are lost, and the YAML is reformatted. Closing ResInsight or the project, and **Rescan Workflows**, ask before discarding unsaved edits. ## Type / format → binding mapping `createBinding()` dispatches in this order — first `resinsight_type` (rips object pickers take priority), then `format`, then bare `type`. Everything unrecognised falls through to the string binding. | Field annotation (Pydantic, in the workflow's task) | Helper emits | C++ binding | PdmField type | Renders as | |------------------------------------------------------------------|---------------------------------------|------------------------------|-------------------------------------|------------------------------------------| | `rips.EclipseCase` / `ObjectModel[rips.EclipseCase]` | `resinsight_type: EclipseCase` | `RimWorkflowCaseBinding` | `PdmPtrField` | Combo box of loaded cases | | `rips.WellPath` / `ObjectModel[rips.WellPath]` | `resinsight_type: WellPath` | `RimWorkflowWellPathBinding` | `PdmPtrField` | Combo box of well paths | | `rips.View` / `rips.EclipseView` / wrapper | `resinsight_type: View` | `RimWorkflowViewBinding` | `PdmPtrField` | Combo box of `Case / View` labels | | `datetime.date` / `datetime.datetime` | `type: string`, `format: date`* | `RimWorkflowDateBinding` | `PdmField` | Calendar widget (`PdmUiDateEditor`) | | `pathlib.Path` / `pydantic.FilePath` / `pydantic.NewPath` | `type: string`, `format: path` / `file-path` | `RimWorkflowFilePathBinding` | `PdmField` (file mode) | Text field + `[Browse…]` (file dialog) | | `pydantic.DirectoryPath` | `type: string`, `format: directory-path` | `RimWorkflowFilePathBinding` | `PdmField` (dir mode) | Text field + `[Browse…]` (folder dialog) | | `bool` | `type: boolean` | `RimWorkflowBoolBinding` | `PdmField` | Checkbox | | `int` | `type: integer` | `RimWorkflowIntBinding` | `PdmField` | Spinbox | | `float` | `type: number` | `RimWorkflowFloatBinding` | `PdmField` | Decimal text field | | `str` (or anything unrecognised) | `type: string` | `RimWorkflowStringBinding` | `PdmField` | Plain text field | \*`format` extraction goes through `pydantic.TypeAdapter(field).json_schema()` so any future Pydantic-native format Pydantic supports is picked up automatically — no hand-maintained map. The helper reassembles `Annotated` forms before asking, otherwise `Annotated[Path, PathType('dir')]` would lose its `dir` marker. `RimWorkflowFilePathBinding` is one class with an `m_selectDirectory` bool toggled at `defineEditorAttribute` time; the same `caf::PdmField` flips between file-open and folder-pick dialogs based on the schema's format. ## Pre-filling from `input.yaml` A binding's initial value comes from, in order: 1. the task's block in `input.yaml` (the definition's `inputs`), or 2. the field's `default=` in the Pydantic model, or 3. the `PdmField` default (empty string, 0.0, today's date, and so on). Bindings are kept with their values when the workflow is edited, as long as the field still has the same kind of binding (`syncTaskInputs`). Values of fields that become wired are kept aside and restored if the field is unwired again. Renaming a task carries its values along. ## The rips object reference round-trip For fields typed as rips classes, the config field schema carries `resinsight_type` but no `default` (the C++ picker provides the value interactively). When the user picks a case in the UI and runs the job, the C++ binding's `toJsonValue()` writes: ```yaml {"load_model": {"case": {"__resinsight_ref__": "EclipseCase", "case_id": 0}}} ``` The reverse side, `rips.taskmaestro_helper.run.resolve_refs` ([run.py](../GrpcInterface/Python/rips/taskmaestro_helper/run.py)), walks the YAML dict and replaces every `{__resinsight_ref__: , …id…}` map with `{"value": }` before handing it to `taskmaestro.JobConfiguration`. The `"value": …` shape is exactly what Pydantic's `ObjectModel[T]` validator accepts, so workflow authors who declare `case: GridCase` get a real `rips.EclipseCase` injected without any custom validator code. ID schemes: | `resinsight_type` | Key in the YAML map | Resolves via | |-------------------|----------------------|----------------------------------------------| | `EclipseCase` | `case_id` | `rips.Project.case(case_id=…)` | | `WellPath` | `well_path_name` | `rips.Project.well_path_by_name(…)` | | `View` | `view_id` | scan `rips.Project.views()` by `.id` | Whitelist lives in [refs.py](../GrpcInterface/Python/rips/taskmaestro_helper/refs.py) — the single source of truth for the rips types the helper recognises and resolves. Adding a new rips-typed picker means: extend the whitelist, add a `RimWorkflowBinding` class, register it in `createBinding()`. ## The run flow 1. The user runs a `RimWorkflowJob` (*Run* in the context menu or the property page). 2. `RimWorkflowJob::runJob()` checks that the gRPC port is live, finds a Python interpreter (`RiaPreferences::pythonExecutable()` → `python3` → `python`), and writes the job's values to `input.yaml` in a temp folder. 3. `RimWorkflow::prepareRunSource()` picks what to run: | Workflow | Command | |---|---| | Installed | `run --registered ` | | Saved, no unsaved edits | `run ` | | Unsaved edits | `run ` with `PYTHONPATH=` | The edit folder is a private temp folder holding the current revision (`save --workflow-only`). The files on disk are not touched until Save. 4. `RiuWorkflowJobRunner` starts the helper with `--input /input.yaml --grpc-port --run-id `. The helper connects to ResInsight, resolves refs, and runs the workflow through `taskmaestro.Runner`. 5. The helper emits newline-delimited JSON events. Task state events update the job and the colours of the graph nodes; everything else goes to the log. Cancel terminates the process. ## Code layout - C++ model: `ApplicationLibCode/ProjectDataModel/Workflow/` (`RimWorkflow*`). - Commands: `ApplicationLibCode/Commands/WorkflowCommands/`. - UI: `ApplicationLibCode/UserInterface/RiuWorkflow*`. - Python helper: `GrpcInterface/Python/rips/taskmaestro_helper/`, tests in `GrpcInterface/Python/rips/tests/test_taskmaestro_helper*.py` (run with `--noconftest`). - C++ unit tests: `ApplicationLibCode/UnitTests/RimWorkflow*-Test.cpp`, `RiuWorkflowGraphLayout-Test.cpp`. Touched (existing files): - `ApplicationLibCode/ProjectDataModel/RimProject.{h,cpp}`: the `workflowCollection` child field, attached at the `MainWindow.Scripts` tab and marked with `disableIO()`. - `ApplicationLibCode/UserInterface/RiuMainWindow.{h,cpp}`: hosts the editor dock and connects graph selection to the property view. - `ApplicationLibCode/Application/RiaGuiApplication.{h,cpp}`: asks before unsaved workflow edits are discarded. ## Constraints - **gRPC server must be on.** The run command refuses to launch otherwise. Set in *Preferences → Python → Enable gRPC server*. - **`PATH` must point at the Python that has `rips` and `taskmaestro` installed**, because `findPythonExecutable()` searches `PATH`. Either launch ResInsight from a shell with the venv on `PATH`, or set *Preferences → Python → Python Executable Location* to an absolute path. - **Rescan to pick up changes on disk.** Use *Rescan Workflows* on the Workflows node after adding folders to `~/.taskmaestro/workflows/` or installing packages. The task catalog is cached until then. - **The collection is not serialised** to `.rsp`. Jobs and their values live for one ResInsight session; save the workflow to keep the Default job's values in `input.yaml`. ## Where to extend - **New rips-typed picker** (e.g. `rips.SummaryCase`): add the class to `RIPS_TYPE_WHITELIST` in `refs.py`, add the resolver branch to `make_rips_resolver` in `run.py`, add a `RimWorkflowSummaryCaseBinding` modelled on `RimWorkflowCaseBinding`, register it in `createBinding()`. - **New scalar/format**: usually nothing on the Python side (TypeAdapter handles it). On the C++ side, add a binding subclass and a one-line dispatch case. - **Change the discovery directory**: edit `RimWorkflowCollection::discoveryDirectory()`. A `RiaPreferences` field + Preferences UI would be a small follow-up if multiple roots are wanted. - **New edit operation**: add a pure function to `RimWorkflowDefinitionTools` with a unit test, call it through `RiuWorkflowEditorWidget::applyEdit`, and add it to the graph context menu in `RiuWorkflowEditorWidget`. - **Supporting nested workflows in the editor**: extend the definition JSON and `RimWorkflowDefinitionNode`, drop the read-only reason in `definition.py` and `structuralReadOnlyReasons`, and teach `save` to write them.