Visual monitoring pipelines¶

The visual-language review defines the shared node, port, connection and state grammar, its current V1 execution profile, and the boundaries an extension must preserve.

The shared Pydantic definition is the public and persisted execution contract. The control plane compiles topology, feature order, lineage, parameter defaults and compatibility without importing numerical packages. Browser wire types are generated by backend/scripts/generate_pipeline_types.py; the component catalog also provides bounded field descriptors. An unsupported parameter schema fails catalog generation instead of falling back to a JSON editor.

Each design has an optimistic-concurrency draft and immutable published revisions. A draft save commits its definition, layout, name and description under one expected version. The editor acknowledges the submitted snapshot while retaining edits made during the request as unsaved work. Publishing freezes that submitted snapshot; later local edits remain in the draft session. A run retains its executable definition, resolved input bindings, initial layout, service identity and predecessor independently of the live service record. Layout updates have their own version and do not change numerical compatibility. Runs also retain the resolved reading, entity and source labels, collection policy, catalogue revision and collection fingerprint. These presentation records do not change execution fingerprints. Historical legacy launches retain their original configuration; a reviewed compatibility normalization is used only when rendering their graph or preparing an explicit release restart.

Each component carries its implementation identity and contract version. Every monitor runs a bound graph (launch version 2). The setup forms submit their configuration, and TACTIC converts it with legacy_definition() into the equivalent graph and activates it like a direct apply. Inputs follow the sorted sensor-key order of the form, and a min-max step in front of a unit-interval detector clips to its range. Monitors saved by the setup forms before they ran as pipelines (launch version 1) are converted the same way at the next release: each restarts as a pipeline and learns again, and one whose readings left the catalogue is stopped instead, keeping its history. Internal image pins and typed MQTT launch overrides are accepted only by trusted activation/release paths, never public monitor creation. Published revisions retain the catalog hash as provenance. Activation recompiles the saved component contracts and requires the saved executable fingerprint to match. Unrelated catalog descriptions or components do not invalidate a revision.

Creation workbench¶

?create=monitoring opens one setup page with the service name, entity, reading bindings and processing graph. Its main actions are Load template, Save template and Apply monitoring. Users do not save drafts or publish revisions to start a service. Learning settings and component parameters stay in the graph inspector. Learning fields appear immediately when its panel opens; only advanced input-slot editing is collapsed. Entity selection includes configured entities even when there is no recent traffic; collection and reading ownership still gate Apply.

Standard setups show a Readings to monitor checklist directly below the entity: uncheck unwanted readings to monitor any subset, such as AC1 and AC3. Select all available and Clear support larger entities; a service accepts up to 100 readings. Selection changes the executable slots and bindings together and retains input IDs across removal, reselection and undo. Saved/custom templates keep explicit named input mappings, with Add reading and Remove on the setup page. Removing an input referenced by a processing component still requires fixing that component before validation and Apply can succeed.

Load template reads the latest saved definition and layout, independently of any historical published revisions. Loading never changes the source template. Template names and descriptions are separate from the service name. A template can be saved without choosing readings or starting monitoring. Subsequent saves use its expected version; a concurrent edit preserves the local work and offers Save as new template. Applied service snapshots do not appear in this library. Migration 013 adds monitoring_pipelines.is_template, preserving all existing designs as templates.

Apply monitoring validates the graph and sends a typed POST /v1/monitors/pipelines/apply request with its definition, layout, bindings, service name and idempotency key. One server transaction creates the immutable execution snapshot, resolves bindings and reserves the existing durable activation. The normal preflight, source claims, capacity checks, worker heartbeat and replacement controller are reused. Retrying the same request returns the same operation. The browser retains uncertain requests across navigation and reload; fields stay fixed until the start finishes or cancellation is acknowledged. An explicit rejection permits corrections; a failed operation offers Return to setup.

Edit service opens its actual running definition, layout and bindings directly. Apply changes clearly explains interruption and fresh learning, then uses the same preflight and durable replacement path. Loading a template or editing settings does not affect the existing service until Apply. History remains immutable. Older library, revision and in-flight activation links remain supported.

Unsaved graphs infer feature metadata with POST /v1/monitors/pipelines/validate, using the same typed, bounded compiler report, administrator authorization and gateway compilation rate budget as saved templates. This endpoint creates no rows. Creation and templates share the same graph controller and account-scoped setup session for bounded history, component insertion, arrangement and validation. Template saves acknowledge only their submitted snapshot and preserve later edits. Leaving for Sources keeps a save alive without navigating the detached screen. Returning joins that operation and restores the setup. A remembered login restored after reload retains the same member's drafts; logout and account changes clear them, and late responses cannot repopulate cleared state.

The shared canvas uses a 16-unit snapping grid, fine blueprint lines and stronger cross markers in both themes. Visible lines become coarser at distant zoom levels while remaining aligned with the underlying snap positions. The magnet control switches drop, drag and keyboard positioning between snapping and free movement. This is a presentation preference: toggling it never rewrites saved positions or execution semantics. Existing layouts move only through explicit user actions.

Creation, saved designs and running graphs share a canvas-first workbench. Add component opens a searchable drag-and-drop menu; service and component settings share one collapsible inspector. Both panels start closed. Mobile overlays the inspector instead of narrowing the graph. Cards use larger titles, stage colors and typed ports; descriptions appear in component settings. Opening and arranging retain readable text size, while Fit entire graph explicitly requests an overview.

Monitoring history¶

The service Evidence tab presents one timeline for its recorded inputs and joint detector result. Readings retain independent scales and units; the raw anomaly score (or logarithmic static evidence) appears once below them. A dashed line shows the threshold, red input lines mark service alarms, and score dots mark alarms from the selected detector. The view works with multivariate models such as online isolation forest and does not infer per-sensor attribution or a baseline.

Up to six input tracks may be displayed simultaneously. Changing that selection only changes the view: results still describe the service's complete input set. Shared hover uses each result's original input timestamps to retrieve its exact recorded readings. Missing input history is labelled unavailable, never replaced with a nearby observation or a synthetic zero. Zoom and keyboard navigation apply to every track and remain fixed during inspection as new data arrives.

Telemetry polls per visible reading, while evidence polls once per input entity with a service filter applied before database pagination. This optional service_id filter is propagated through the gateway chart-evidence endpoint; existing unfiltered consumers and cursor ordering retain their contract. History uses the signed-in workspace's bounded cache (2,000 retained points per displayed input and 2,000 result rows). Ordinary reading charts remain independent of a monitor; when monitoring is present their alarm markers and results share the same service scope.

Numerical ownership¶

Aberrant 1.3 supplies the standalone aberrant.compose.GraphPipeline numerical API. OFF-KEY compiles into that API and owns bindings, lifecycle, calibration, alarm policy, persistence and presentation. Aberrant has no dependency on these application services or wire formats. Existing linear Aberrant pipelines retain their existing behavior.

Adaptive graphs learn each shared node once in topological order. Only ready outputs reach successors, including on the observation that first establishes readiness. Detector warm-up counts successful terminal updates. Calibration starts at the next scoring opportunity after both that target and actual readiness are satisfied. Calibration and monitoring score before learning. Retained-value readiness can wait for suitable data without retaining an unbounded input buffer. Generated constructor schemas also encode retained-window capacity constraints from the pinned runtime's parameter-sensitive capabilities. For example, geometric averages require two retained values, or three with absoluteValues=true, while harmonic averages permit a one-value window. Catalog generation verifies these rules without introducing numerical imports into the control plane.

Static graphs fit each preprocessing component on training observations only. They freeze before transforming calibration data, fitting conformal calibration, or monitoring. Fitted preprocessing, detector and policy publish together after background training. Static centered batch PCA and adaptive uncentered incremental PCA have different implementation identities. Scalers and random projections also have strategy-specific identities. Changing preprocessing coordinates online does not re-express geometry already retained by an adaptive detector.

Raw branches, selection and explicit renaming retain lineage. Ordered merges join branches from the same aligned observation. Name collisions fail validation; projection features are scoped by their node ID. Selection uses stable slot IDs, while resolved reading topics remain the evidence identity. Expected units must match exactly; the application performs no implicit conversion. Incomplete drafts still expose every knowable feature schema. An invalid node blocks inference for its dependent paths while independent branches and valid upstream outputs remain available to the inspector. Executable compilation still requires the entire graph to validate.

Activation and recovery¶

Activation persists reservations before remote work. Claims cover the union of predecessor and successor inputs and reserve one successor capacity slot. Every form start, graph activation, collection mutation and release consults this authority under the collection configuration lock.

The controller runs credential-free constructor preflight in the exact target RADAR image digest before retiring a predecessor. The report binds the executable, catalog, collection and bindings to that digest. It reports known numerical allocation costs; it does not prove data-dependent fitting or predict every model allocation. Preflight has separate workload identity, bounded concurrency and timeout, and cleanup ownership. A fresh nonfailed heartbeat confirms activation; warm-up remains a run stage. Activation supervision uses separate bounded capacity for preflight, remote transitions and database checks, each with independent sessions and durable leases. Orphan cleanup runs separately. Configure constructor capacity with TACTIC_PIPELINE_MAX_CONCURRENT_PREFLIGHTS (default 2); remote transitions and checks each use TACTIC_PIPELINE_MAX_CONCURRENT_TRANSITIONS (default 8). Oldest updated operations are considered first. The database records the first healthy heartbeat receipt, preserving timely startup evidence across later heartbeats and delayed controller checks. A first heartbeat received after the deadline cannot establish successful startup.

Remote uncertainty retains reservations until shutdown is confirmed. Cancellation before retirement preserves the predecessor. Cancellation after retirement creates no implicit rollback. Stopping a predecessor also cancels its pending replacement. Reconciliation resumes durable operations after controller restart. The durable sequence is requested, preflighting, validated, retiring, starting, awaiting_heartbeat, then succeeded; cancellation/failure uses cancelling until cleanup is confirmed. A controller token and expiring lease fence every resumed phase and error transition after remote I/O. Stale replies cannot move a newer activation back to an earlier phase.

Terminal lifecycle transactions lock checkpoint, run and service records in that order, including service deletion's checkpoint cascade. Heartbeat acceptance and fatal-run recording serialize their run/service updates so a stale healthy heartbeat cannot overwrite a committed fatal state. Collection stops, deletion and reconciliation share complete_stopped_runs beside the lock helper. It retains failed/cancelled outcomes, error details and the first terminal timestamp across repeated cleanup.

Deployment reserves maintenance before quiescing writers. It refuses a pending activation, and active maintenance prevents creation and handoff in both HTTP and background paths. Cleanup remains permitted. Release snapshots preserve graph definitions, bindings and run lineage; a software upgrade creates a fresh run when numerical compatibility requires it.

An uncertain mutation crosses resilience and message processing as a fatal error. The worker stops graph execution, blocks checkpoint export including shutdown, preserves the last signed snapshot and records failed-run coverage. That failed generation cannot be admitted again. A graph revision rollback starts a fresh run; same-run task recovery restores only compatible signed state.

Bounded inspection¶

Diagnostics observe the actual serial scoring and learning passes. They perform no additional inference, MQTT subscription or Docker inspection. At most one observation envelope is sampled per second and retained, with eight input/output values per node. Unexecuted nodes carry a reason and never reuse previous values. The serialized snapshot is capped at 128 KiB, with status and counters taking priority over previews. Static fitting appears as a batch summary. Adaptive nodes also report committed update counts, observed/required history in the component's own units, and their immediate blocking prerequisites. A zero required count means no quantified minimum was supplied; the ready flag remains authoritative. Committed counts include restored history, while collector call counts can restart after recovery. Readiness reasons and errors retain at most 128 and 256 UTF-8 bytes respectively, with the same ceilings applied after JSON escaping.

The ownership-fenced heartbeat publishes diagnostics every five seconds during healthy operation. One member-scoped workspace request returns service summaries and optional selected-run diagnostics. The browser pauses hidden work and displays snapshot age separately from evidence persistence freshness.

V1 admits at most 64 executable nodes, 128 connections and 100 features at any intermediate boundary. Bound executable configuration has a 96 KiB serialized budget to stay below the worker environment transport limit. Known allocation admission and existing worker CPU/memory hard limits apply independently. Adaptive PCA admission includes each initialization row's NumPy array header, the old and new pointer lists retained during a learning transaction, the dense initialization matrix and reduced SVD outputs. These costs use the target worker's Python/NumPy object sizes; per-projection peaks are summed conservatively. The estimate also covers explicit online update matrices. Detector state, native linear algebra workspaces and other process allocations remain additional costs, so passing admission does not promise that every workload fits the hard limit. External validation reports, including read-only run metadata, have a 512 KiB serialized budget. Large reports omit provenance first, then whole node tables, with an explicit metadata_budget notice and features_complete / provenance_complete flags. A selected_node_id on draft validation or the run query preserves complete ordered feature names for that node and its immediate parents; feature choices are never truncated. Unknown selections return an explicit issue and no stale metadata. Invalid oversized definition echoes can be omitted separately, and individual errors are bounded to 1 KiB of UTF-8 text. The compiler rejects normalized definitions above 96 KiB before publication; bindings must still fit the complete execution budget at activation.

Library lists return summaries with server-side search, archive filtering and pagination (limit, offset; default 25, maximum 100). Design details include at most 25 revision summaries. Revision pages use before revision numbers; individual immutable definitions load on selection. Run history pages contain summaries and a next_service_id continuation, resolved in one depth-bounded recursive query per page. History never repeats executable definitions or binding snapshots. A selected run includes its design ID so opening its draft needs no library scan. /pipeline-metadata and immutable revision /metadata requests return only the selected node's and its parents' compiler tables, without run history or another definition echo. These member-readable inspections remain available with authoring disabled.

The gateway assigns compilation and automatic metadata inspection a separate member-scoped budget of 120 requests per minute. They cannot consume the 30-per-minute mutation budget or the 120-per-minute workspace polling budget.

Performance evidence¶

backend/scripts/benchmark_pipeline_runtime.py compares real numerical graph execution with observation disabled and enabled, without MQTT, PostgreSQL or health-publication I/O. Both runners receive identical seeded observations and must produce exactly identical scores and alarms. Short, alternating batches of ten observations balance execution order and slow host/VM speed changes within each trial. Whole-run blocks initially showed large speed drift inside identical forest kernels; their timings are not used to infer observation overhead.

The 2026-10-16 paired probe used an AMD Ryzen 5 7530U, WSL2 Linux x86-64, Python 3.14.8 and Aberrant 1.3.0. Docker limited the process to one CPU (cpu.max=100000 100000) and 1 GiB; twelve logical CPUs were visible. OPENBLAS_NUM_THREADS, OMP_NUM_THREADS, MKL_NUM_THREADS and NUMEXPR_NUM_THREADS were all 1. Each of three trials used 512 warm-up observations, 32 calibration score/learn calls and 100 timed observations. The collector used its normal one-second sampling interval, including committed-update and readiness metadata. The report records hardware, thread controls, cgroup bounds, parameters and raw trial timings.

The representative graph has eight inputs and nine nodes: six selected features pass through online standard scaling and three-component uncentered PCA, two features bypass processing, then both branches merge into a seeded 100-tree online isolation forest (window_size=512, max_leaf_samples=16, n_jobs=1). Its frozen quantile policy and output are included. The cheap comparison uses one input, standard scaling and a moving-average detector with a 128-value window.

These are medians of aggregate trial timings. All three paired representative trials measured lower CPU time with observation: no positive cost was measurable at this probe's precision. Those negative differences reflect residual host variation, with no claimed numerical speed improvement. Individual trials without observation ranged from 42.763 to 182.347 ms/observation, so this provides scoped acceptance evidence rather than a precise overhead estimate. The <=10% target applies to this documented representative workload. Cheap detectors can have much larger relative overhead; their measured absolute CPU increase was 0.110 ms/observation. Absolute throughput varies with host load, and these measurements are not an end-to-end throughput guarantee.

Two separate maximum-size fixtures ran 4,000 scored/learned observations each after warm-up. One has 64 nodes, 128 edges and 100 source features with shared branches and reconvergence; the other carries all 100 features through a 64-node, 63-edge chain. Every observation was captured for this retention probe, a stronger allocation load than normal sampling. Both retain stable node keys, eight preview values per side and one observation envelope.

Retained sizes count collector-owned reachable Python objects once, excluding the numerical models and allocator overhead. Snapshots stay below 131,072 bytes. The memory capture, sampled during the final fixture, recorded a cumulative whole-container cgroup memory.peak of 147,513,344 bytes (140.68 MiB), including all workloads up to that point; this is not a per-fixture measurement. Separate maximum-node tests combine 100-character IDs, 128 dependency links, multibyte/control-character readiness reasons and errors, bounding both UTF-8 and JSON-escaped text. The fixture tests also verify one learning update per eligible node and exact observation-on/off score equivalence. Timing-only reruns use --timing-only; a full run also repeats the sustained bounds probes.

The runtime evidence preserves the raw trials, component parameters, environment and retention results. Separate browser evidence records editor rendering checks.

Rollout¶

The production rehearsal restores the current release's backup, runs an older pipeline-compatible build, then returns to the current build. Each cutover checks the actual application and worker image digests, fresh graph scoring, bindings, layout, retained run history and single-worker ownership with authoring disabled. The test-only reference is immutable commit d63f444dbaaf8dec0392dd723394452f2e32fe69, which includes the direct-apply backend and the corrected migration 014, as well as parameter-sensitive readiness constraints and bounded inspection APIs. Its migration ledger, numerical catalog and locked Aberrant version must match the current source before it can build. Other locked dependencies may differ, as they would in a real rollback: each release calibrates its own run, and detector checkpoints refuse a changed runtime. This fixture is not an eligible production release. Changes to those contracts require review of the rollback reference. CI retains nonsecret source/image identities and the asserted graph invariants as its compatible-binary-rollback artifact.

The production rollback rehearsal passed both integration tests with zero skips in 625.685 seconds on 2026-10-16. Its retained artifact records all three cutovers and verifies the deployed image digests and graph invariants; the following browser phase is a separate gate.

The initial compatibility release shipped with authoring off. The visual workbench is now enabled by default with TACTIC_PIPELINE_AUTHORING_ENABLED=true; numerical, real infrastructure, lifecycle, browser and resource gates must pass before release. An explicit operator override to false is preserved. Disabling authoring blocks new designs and workbench activations while the setup forms, existing graph runs, inspection, stopping and release recovery remain supported.

The software rollback floor is a pipeline-compatible release. Older binaries cannot interpret the migration ledger or graph configurations. Use a compatible binary, or the reviewed release database backup/restore procedure. Never erase migration history to make an old binary accept a newer database.