09 Operations Observability

Operations and Observability

Status: IMPLEMENTED CORE / ACTIVATION GATED. Six deterministic source-refresh schedules and one freshness schedule are registered stopped. Source schedules inject explicit live resource config and exclude paid inference. The production reference stack has durable Dagster storage, one daemon, a queued coordinator, non-root images, and blank/versioned schema preparation; a disposable full-stack acceptance passes. Live-source operation, natural partitions, external alert transport, backup/restore rehearsal, consumer cutover, and corpus-wide atomic query publication remain unproven. Clinical trials is generation-protected and failure-injection tested. Denoising state: sharp (noise -> rough -> sharp -> converged) Updated: 2026-07-25

Purpose

Define how the system runs, observes itself, recovers, and alerts. Dagster is the orchestration + observability plane; the curation store is observed, not owned, and native inference-batch providers are external asynchronous systems.

Automation policy (decided)

• Prefer declarative automation for deterministic asset conditions that need no launch-time resource config. Use an explicit scheduled asset job when a source cadence must bind audited live resource config and an exact root selection. Both are native Dagster asset automation, not a return to task orchestration. (https://docs.dagster.io/guides/automate/declarative-automation, https://docs.dagster.io/guides/build/jobs/asset-jobs)
• A source schedule selects exact source roots plus deterministic descendants, excludes all other source roots and every kind="llm" asset, and carries bioloupe/refresh_kind=deterministic.
• The two deliberate sensor exceptions are external async systems: inference batch polling uses persisted provenance.inference_batch_jobs handles, and reviewed curation reactions use a cursor watermark + deterministic run_key. Both launch normal runs for data-changing work. (https://docs.dagster.io/guides/automate/sensors)

Source Cadence (target and current boundary)

Six source cadence schedules and one operational freshness schedule are loaded into Definitions; all are stopped by default. The source schedules override their owning resource to explicit live mode with no max_live_records, while the general code-location resources remain fixture-default for development/tests. BIOLOUPE_RUNTIME_ENV=production forbids fixture capture, but it does not turn an unscheduled/manual source family live. Source assets are currently unpartitioned. The target is source-native windows/releases only where a source can replay them deterministically; APIs exposing only current state retain append-only snapshots and must not receive fake historical partitions.

Schedule	Live root and boundary
ops_ctgov_incremental_daily	CTGov discovery/snapshots; requires one explicit complete bootstrap and watermark first
ops_fda_regulatory_weekly	complete FDA approval plus REMS/CDx refresh
ops_identity_authority_missing_daily	missing-only NCIt/RxNorm/MeSH authority lookups
ops_target_authority_missing_daily	missing-only HGNC/ChEMBL target evidence
ops_clinical_vocabulary_monthly	complete official CTCAE workbook plus project endpoint catalog
ops_epidemiology_weekly	complete configured CDC dataset
ops_ingestion_freshness_hourly	database-only observability refresh; no source or provider call

ChiCTR, conferences, Crossref, FMP, news, PubMed, SEC, and FDA-label source groups require an operator-owned cohort/import/credential/feed decision and intentionally have no generic schedule. The versioned Bioloupe taxonomy has an explicit deterministic refresh job but no cron.

flowchart LR
  Cron[Loaded schedule, stopped by default]
  SourceAssets[Bounded source assets, currently unpartitioned]
  Checks[Blocking checks]
  Canonical[Canonical assets]
  Query["q_* read models"]
  Store[[Reviewed curation store]]
  Sensor[Curation sensor with cursor + run_key]
  BatchLedger[[provenance.inference_batch_jobs]]
  Provider[[Inference batch providers]]
  BatchSensor[Inference batch sensor]
  Alerts[Alerts / runbooks]

  Cron --> SourceAssets --> Checks --> Canonical --> Query
  Store -. observed .-> Sensor --> Checks
  SourceAssets --> BatchLedger
  BatchLedger -. persisted handles .-> BatchSensor
  Provider -. status/results .-> BatchSensor
  BatchSensor --> Checks
  Checks -- failure --> Alerts
  Sensor -- stalled watermark --> Alerts
  BatchSensor -- stalled handle --> Alerts

External asynchronous operations

Two external systems are deliberately observed rather than owned by Dagster: inference batch providers and the reviewed curation store. Both use persisted state plus sensors so recovery does not depend on a long-running process.

Inference batch lifecycle

sequenceDiagram
  autonumber
  participant Asset as Evidence/resolution asset
  participant Lifecycle as provenance.inference lifecycle
  participant Provider as Native batch provider
  participant Sensor as Dagster batch sensor
  participant Run as Dagster ingestion run
  participant Downstream as Evidence/resolution outputs

  Asset->>Lifecycle: atomically claim preparing cohort + persist pending attempts
  Asset->>Provider: submit batch
  Provider-->>Asset: provider batch handle
  Asset->>Lifecycle: store profile snapshot + handle
  Sensor->>Provider: poll handle
  Provider-->>Sensor: running/completed/failed/expired
  Sensor->>Run: RunRequest with stable run_key on completion
  Run->>Provider: fetch results
  Run->>Lifecycle: attempts + findings + decisions
  Lifecycle-->>Downstream: accepted-result view only

Operational invariants:

• never hold a Dagster run open while waiting for provider batch completion;
• batch state is recoverable from the DB: request ids, exact contract/profile snapshots, provider handle/status, timestamps, and result counts;
• ingestion matches provider results by custom_id, never by provider-returned order;
• the sensor only polls state; payload fetch/validation/persistence runs in inference_batch_ingestion with Dagster retries and observability;
• lifecycle ingestion never writes slice evidence; a batch-capable slice owns a later projection/rematerialization from accepted results;
• the submission claim is stable across retries of one Dagster run/asset and distinct across deliberate re-executions;
• submission_unknown is never auto-resubmitted; reconcile the provider by the local batch/request metadata before attaching a recovered handle or launching a new cohort;
• failed/expired/cancelled batches remain terminal evidence; recovery creates an explicit new attempt cohort rather than mutating them;
• cost and token usage are normalized on attempts even when providers expose different usage fields.

Reviewed curation lifecycle

sequenceDiagram
  autonumber
  participant Agent as Curator agent
  participant Human as Human reviewer
  participant Store as curation.decision_events
  participant Sensor as Dagster curation sensor
  participant Checks as Review + publication checks
  participant Query as q_* read models

  Agent->>Store: propose correction/suppression with evidence_refs
  Human->>Store: approve/reject high-impact proposal
  Sensor->>Store: read decisions after cursor where review_state is publishable
  Sensor->>Checks: launch owner-routed canonical job with deterministic run_key
  Checks-->>Query: trials promote atomically; other families still publish in place

Operational invariants:

• curation is not a hidden write to canonical/query tables;
• pending high-impact proposals are visible in review metrics but do not publish;
• reviewed decisions carry stable typed subjects so owner routing is deterministic;
• the database-owned event cursor and run_key make retries idempotent;
• owner jobs currently rebuild a slice, not one row/partition.

Backfills (decided)

Natural partitions are the target for replayable source windows and large inference cohorts. Current source assets are unpartitioned, so operators must use explicit bounded resource config and verify the requested window; whole-slice reruns are still a known operational limitation. (https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets)

Legacy Data Gov salvage is the first cutover-specific backfill family: read-only legacy rows are staged in migration.* tables, mapped legacy LLM/review outputs become provenance-bearing evidence/curation inputs, and fresh LLM backfills run only for remaining gaps. See 11-legacy-salvage-and-backfills.md for the exact staging/backfill/salvage contract.

Failure semantics (decided)

• Assets fail loudly; no swallow-and-continue. Narrow, intentional retries (with backoff) only for known-transient source/LLM errors.
• For clinical_trials, candidate construction, blocking validation, all stable-view replacements, serving-head update, and promotion receipt share one transaction. A failure leaves the previous generation served; inspect ops.query_publication_heads and ops.query_publication_events.
• Other query families are rebuilt in place. A failed blocking check stops later selected assets but cannot prove that their prior served relation survived; disable the affected path and inspect/restore it.
• Before a release or foundational graph change, make verify-bootstrap must prove the entire fixture graph and all checks can build from an empty database without provider keys, network capture, legacy DB access, or pre-existing relations, then pass P01-P12 through checked-in SQL probes on a forced read-only connection.

Observability + lineage/provenance (decided)

• Every materialization emits structured metadata (row counts, model/prompt/schema/provider versions, LLM batch handle where relevant, provenance bundle counts, immutable source artifact pointers, freshness) as asset metadata, surfaced in the Dagster catalog.
• For external lineage interchange, align with OpenLineage’s run/job/dataset + facets model. (https://openlineage.io/docs/spec/facets/)
• LLM throughput/cost/quarantine rate and curation review throughput/queue age/conflict rate/override-reversal rate are first-class observed metrics.

Source Cadence and Alert Matrix

Source family	Automation shape	Freshness/alert pressure
Repository-versioned taxonomy releases	reviewed source-file + active-release change; explicit materialization, no cron	release-id/content immutability, active-pointer drift, failed graph rebuild
Registries / regulators	loaded stopped schedules; unpartitioned bounded runs today; source-native replay windows are the target	freshness SLA, source schema drift, count delta
Labels/documents/publications/news/filings	loaded stopped schedules; fixture/import/live capability varies by owner; document snapshots persist exact versions	artifact version drift, parser/chunker failure, source locator missing
LLM batch providers	submit/poll/ingest sensor with persisted handles	stuck batches, expired/cancelled batches, validation failure rate, cost anomalies
Agentic curation store	sensor with cursor + deterministic run_key over reviewed decisions	watermark stalled, review queue age, conflict rate, rejected/pending high-impact overrides
Downstream support/readiness models	derived from canonical/query assets after checks pass	stale readiness, missing variable dictionary rows, downstream-boundary creep
Search/discovery projections	refresh after upstream content/query changes	stale chunks/embeddings, mixed embedding dimensions, invalid query recipes

Alerts

Alert on: source freshness SLA breach, source schema/parser drift, missing source-artifact immutability/provenance, blocking-check failure on a publication path, LLM batch stuck/expired, curation review queue aging beyond threshold, sensor non-advancement (watermark stalled), embedding/search projection drift, and downstream-support readiness regressions.

Runbook Matrix

Symptom	First check	Likely owner	Safe recovery action
Source count drops to zero	raw source snapshot materialization metadata and source HTTP/API status	source family package	disable the affected schedule, inspect whether in-place query state changed, and restore the last verified backup if necessary.
Parser/schema drift	source artifact version + extractor/schema version metadata	source/evidence package	quarantine new evidence claims; update the owning source/evidence contract and use an ADR only when the architecture changes.
Blocking check fails	asset-check result plus publication_protected; for trials inspect generation head/events	relevant slice + contracts	protected trials retain the serving head; fix and rerun. For an in-place family, disable the path and inspect/restore the relation.
Trial query rollback requested	target generation manifest, current/previous head, promotion events, reason	clinical-trials query owner	call rollback_trial_query_family for a retained complete generation; it verifies the current contract versions and mutation guard, repoints all seventeen stable views, and records one rollback event transactionally.
Inference batch sensor stalls	provenance.inference_batch_jobs, provider status, last poll timestamp, ingestion run history	inference runtime owner	resume polling the persisted handle; after expiry/cancel, launch an explicit new attempt cohort for unresolved requests.
Inference batch submission unknown	local batch id/request metadata, provider batch listing, provider audit logs	inference runtime owner	attach the exact recovered provider handle; only after proving no provider batch exists, mark failed and launch a new execution cohort.
Curation sensor stalls	sensor cursor, last reviewed decision id, run keys	curation package	resume from the last committed sensor cursor and re-emit deterministic run_keys; do not mutate curation.decision_events; idempotency keys prevent duplicate decision rows and run keys prevent duplicate rematerialization runs.
Search projection stale	source row updated timestamp vs search projection timestamp	discovery/search package	rematerialize affected search docs/chunks only.
Mixed embedding dimensions	embedding metadata model/version/dimensions	discovery/search package	quarantine affected vector rows; rebuild index after model migration plan.
Forecasting readiness regression	readiness model provenance and missing variable dictionary rows	downstream-support/query package	publish current readiness; downstream app handles stale saved scenarios.
Blank-database acceptance fails	failed Dagster step or P01-P12 scenario id plus compact evidence from make verify-bootstrap; rerun with --verbose --keep-database	asset/table owner named by the first failed step or scenario owner in 02	repair the owning dependency or contract; never add an ordering sleep, downstream table-creation workaround, or weaker scenario threshold.

Activation Contract

A healthy Compose stack proves packaging and topology, not safe live automation. Turn on one path at a time only after its row below is satisfied; do not enable all schedules as a single launch event.

Gate	Required evidence before enabling affected automation
Repository	make check, make verify-bootstrap, installed-wheel definition validation, locked gateway/web tests and build; gateway trusted-document mismatch and malformed-envelope tests pass
Database	prepare-schema against the target predecessor, backup taken, restore rehearsal passed, revision/head recorded
Source	bounded real-source smoke with contract/hash/provenance checks, then an explicit complete bootstrap/window; no bounded smoke may become the active complete cohort
Publication	candidate-before-promotion last-known-good protection, or an explicitly approved maintenance-window/restore procedure for that exact in-place family
Inference	no source schedule can call a model; each separately enabled paid task needs a passing provider-backed semantic eval on the pinned corpus/profile plus an explicit retry-inclusive budget
Operations	external alert destination tested for run failure, freshness breach, stuck batch, and stalled curation cursor; daemon heartbeat and queue observed
Consumer	read-only SQL role, representative query/latency smoke, shadow comparison, named consumer owner, rollback/cutover decision

Current activation verdicts:

Path	Verdict	Blocking evidence
Production stack, schedules stopped	locally proven	Full disposable Compose boot, blank install to Alembic head, second install no-op, loaded definitions, non-root runtime, gateway health, and unauthenticated GraphQL denial pass. This is not a live-host rehearsal.
CTGov deterministic schedule	closest, still gated	Run bounded live smoke, explicit initial complete bootstrap/watermark, backup/alerts/consumer shadow. Its seventeen-model query family is atomically protected.
FDA regulatory schedule	not unattended-ready	Run live smoke/complete bootstrap and add atomic approvals-family publication or approve/test a maintenance recovery procedure. Non-FDA live breadth remains absent.
Identity, target, vocabulary, epidemiology schedules	not unattended-ready	Run each real-source smoke/bootstrap and protect every affected query family (or approve/test exact recovery).
Manual source groups	operator-only	Supply exact cohort/import/feed credentials and source-specific completeness proof; no generic schedule exists by design. FDA labels specifically require an exact application cohort or bounded query, and incomplete capture cannot change the active selector.
SOC publication	live breadth blocked	Labels produce regulatory approved-use evidence, not SOC. Live compound-label decomposition and a lawful/versioned professional-guideline source are absent; only an explicit human-approved promotion can currently create an SOC assertion.
SOC, trial eligibility, target-candidate inference jobs	off	Provider-backed semantic evals have not been run. Default-off typed gates, cohort caps, retry-inclusive preflight ceilings, and deployment concurrency protect against accidental spend. The existing SOC task only fills missing line/setting on pre-existing authored candidates; it is not live label decomposition.

Remaining product-wide stop conditions are not code-location health problems: China/ChiCTR live capture and several non-FDA/guideline/conference source families are incomplete; most query families still publish in place; no production backup/restore or alert path is proven; no downstream consumer has completed shadow/cutover; and paid semantic inference quality has not been measured.

Deployment Choices

• Alert transport is deployment configuration: Dagster+ alerts or OSS hooks may carry the same named conditions without changing pipeline semantics.
• The Dagster catalog is the required local lineage surface. An OpenLineage emitter is optional interoperability and must not become a second provenance authority.
• The checked-in single-host reference deployment runs one internal webserver, exactly one daemon, a queued run coordinator, and PostgreSQL-backed run/event/schedule storage in a database separate from the Bioloupe data database. A one-shot Alembic service must bring the data schema to head before either Dagster process starts. The authenticated gateway is bound to host loopback for an operator-owned HTTPS reverse proxy; Dagster and PostgreSQL have no host ports. The gateway requires exact HTTPS origins, canonical-AST trusted GraphQL documents, bounded JSON bodies, and read/operate authorization. Runtime application services use fixed unprivileged UID/GID 10001; application code is not writable. Maximum concurrent Dagster runs and each default pool limit are 1 until workload-specific capacity and publication isolation are proven.
• This reference topology is implemented and full-stack tested against a disposable blank database; it is not evidence that production backup/restore, failover, alerts, or live schedules have been exercised.