04 Dagster Asset Graph

Dagster Asset Graph

Status: DECIDED / PARTIAL RUNTIME. Asset graph, provenance, metadata, and naming conventions are implemented. Clinical trials has protected publication generations; natural source partitions and adoption by other query families remain open. Live/consumer state: fixture-proven only; no production or consumer cutover evidence. Denoising state: sharp (noise -> rough -> sharp -> converged) Updated: 2026-07-24

Purpose

Define the asset graph conventions for the whole system: what counts as an asset, how assets are grouped and named, partitioning, mandatory metadata, and the boundary between pipeline lineage and domain provenance. Slice docs supply concrete asset keys; this doc supplies the shared grammar.

Assets are durable nouns

An asset is a persistent object (table/view/model); an asset definition is the code that produces/updates it. (https://docs.dagster.io/guides/build/assets) Every asset in this repo is a noun in the layered graph below — never a task verb.

Layered graph (decided)

flowchart LR
  subgraph source_group["source_<source> group"]
    SourceSystem[[External source system]]
    Raw["sources.<source>.*\nraw snapshots"]
  end

  subgraph normalized_group[normalized group]
    Normalized["normalized.<entity>\nsource-faithful records"]
  end

  subgraph evidence_group[evidence group]
    Evidence["evidence.<claim>\nclaims + provenance"]
  end

  subgraph resolution_group[resolution group]
    Resolution["resolution.<entity>\nidentity candidates"]
  end

  subgraph curation_group[curation group]
    CurationStore[["curation.*\nobserved override/review store"]]
  end

  subgraph provenance_group[provenance group]
    Provenance["provenance.*\nsource artifacts + inference lifecycle + assertion links"]
  end

  subgraph canonical_group[canonical group]
    Canonical["canonical.<entity>\naccepted facts + contextual synthesis"]
  end

  subgraph query_group[query group]
    Query["query.q_*\nagent SQL read models"]
  end

  SourceSystem --> Raw --> Normalized --> Evidence --> Resolution --> Canonical --> Query
  CurationStore -. observed external asset .-> Canonical
  Raw --> Provenance
  Evidence --> Provenance
  Resolution --> Provenance
  Canonical --> Provenance
  CurationStore -. reviewed overlay receipts .-> Provenance
  Provenance -. q_provenance receipts .-> Query

Each arrow is a Dagster asset dependency, forming the data lineage Dagster renders natively. (https://docs.dagster.io/guides/build/assets)

Cross-slice synthesis stays inside the graph

Therapeutic development programs, epidemiology series, and commercial summaries do not justify a new pipeline layer. They are canonical or query assets with several canonical dependencies:

canonical owner facts -> canonical contextual synthesis -> query.q_*

They never depend on query.q_*. Query models are terminal consumer contracts, not convenient upstream integration tables. A synthesis asset must preserve the owner keys and provenance bundles of every input fact, declare its own grain and derivation revision, and leave source-specific truth with the producing slice.

Asset key + group conventions (decided)

• Key prefix = layer.concept, e.g. ["sources","ctgov","trial_snapshot"], ["evidence","trial_design_claims"], ["query","q_trials"].
• Group name = layer (source_<src>, normalized, evidence, provenance, resolution, curation, canonical, query) so the UI groups by pipeline stage.
• Kinds/tags record compute kind: python, sql, llm, external (inbound ingestion), external_asset (observed curation override/review store). llm assets call the shared inference runtime from 06-llm-extraction-and-verification.md; native batches split submit/poll/ingest across observable Dagster runs.
• Requests, attempts, findings, decisions, and provider batch state are one durable provenance noun: ["provenance", "inference_lifecycle"]. Individual model rows are Postgres records, not Dagster assets or partitions.
• Owners tag the responsible domain slice.

Dagster supports attaching metadata, tags, owners, group names, and kinds to assets; this repo treats those annotations as part of the documentation surface, not UI decoration. (https://docs.dagster.io/guides/build/assets/metadata-and-tags/)

Physical relation ownership

• Every checked-in DDL file has exactly one Dagster owner asset. A downstream writer declares that owner as a dependency and writes rows; it does not recreate the table as defensive setup.
• Source repositories may initialize only their own sources_* relations. Shared provenance, evidence, resolution, canonical, identity, and ops relations are initialized by their layer owners.
• Independent providers get independent source DDL when they write independent tables. When several logical source assets intentionally share one physical store, a small store asset owns the DDL and every capture asset depends on it.
• evidence/trial_claim_tables is the reference shared-store owner: deterministic extraction, legacy salvage, and optional eligibility inference all write trial evidence, but none of those writers independently owns or recreates the relations.
• evidence/soc_claim_tables applies the same rule to deterministic SOC claims, legacy SOC salvage, immutable line/setting projections, and the source-wins effective view.
• tests/test_source_schema_references.py enforces these rules statically. make verify-bootstrap then proves that the declared graph can build every relation from an empty database.

This rule is about ownership, not migration ceremony. DDL may remain idempotent, but IF NOT EXISTS is not a substitute for a dependency and must not be used to hide an undeclared upstream relation.

Spend-safe optional inference

An asset marked llm is not permission to spend. Every provider-calling domain asset owns a typed run config with enabled=false, a finite hard cap, and an optional exact business-key cohort. Disabled execution returns before reading domain tables or constructing the inference runtime. The asset still remains visible in the graph so downstream lineage is truthful; selecting a downstream asset without explicit run config cannot call a provider. Deployment-wide API pressure is controlled separately by the llm concurrency pool.

Canonical key examples

Layer	Example asset key	Group	Kind	Notes
source snapshot	["sources", "ctgov", "trial_snapshots"]	source_ctgov	python / external	Raw source payload or snapshot table; source-faithful.
document artifact	["sources", "dailymed", "label_artifact"]	source_dailymed	python	Versioned label/document snapshot with hash/locator.
normalized	["normalized", "clinical_trials"]	normalized	python	Source-faithful typed records; not effective/canonical truth.
evidence	["evidence", "trial_outcome_claims"]	evidence	python / llm	Source-grounded claims with locator/extractor version.
provenance source artifact	["provenance", "source_artifacts"]	provenance	python / sql	Immutable artifact registry: URI, hash, source version/release, observed/fetched timestamps.
provenance extraction run	["provenance", "extraction_runs"]	provenance	python / sql	Extraction/verification activity metadata: extractor, prompt/schema/model version, run id.
provenance assertion links	["provenance", "assertion_links"]	provenance	python / sql	Receipt links from canonical/query facts to artifacts, extraction runs, evidence, resolution/verifier receipts, and curation overlays when applicable.
inference lifecycle	["provenance", "inference_lifecycle"]	provenance	sql	Provider-independent requests, immutable attempts, validation findings, append-only decisions, accepted-result view, and durable batch state.
resolution	["resolution", "drug_intervention_candidates"]	resolution	python / llm	Candidate links from source intervention names to canonical drug or NCIt concept, owned by 29-slice-drug-identity.md.
curation observed	["curation", "observed_decisions"]	curation	external_asset	Agent-proposed/human-reviewed external curation overlay log observed by Dagster.
canonical	["canonical", "drugs"]	canonical	python / sql	Canonical drug identity spine: substances, aliases, external ids, brands, combinations.
canonical	["canonical", "trial_outcomes"]	canonical	python / sql	Effective quality-passing assertions, with curation overlays applied only when reviewed.
canonical synthesis	["canonical", "development_program_state_history"]	canonical	python / sql	Deterministic contextual state derived from accepted trial, approval, identity, and organisation facts; owned by 33-slice-development-landscapes.md.
canonical observation	["canonical", "epidemiology"]	canonical	python / sql	Multi-table asset publishing full-grain disease-burden observations and comparable series, owned by 34-slice-epidemiology.md.
canonical observation	["canonical", "product_sales_observations"]	canonical	python / sql	Source-reported commercial facts owned by 35-slice-commercial-intelligence.md.
query contract metadata	["ops", "query_contract_metadata"]	ops	python / sql	Sole writer of ops.query_contract_columns, regenerated from checked-in YAML contracts for q_catalog.
query	["query", "q_trials"]	query	sql	Agent-facing read model in query.q_trials.
query catalog	["query", "q_catalog"]	query	sql	Generated catalog over database schema/comments and checked-in contract metadata.
query provenance	["query", "q_provenance"]	query	sql	Agent-facing explain surface for provenance-required rows/facts.

Source and external assets (decided)

• Upstream source systems and the agent/human-reviewed curation store may be modelled as external/observable assets only when Dagster needs to observe state it does not materialize. Dagster’s external-asset docs are explicit that external assets are not directly materialized or scheduled by Dagster. Therefore ClinicalTrials.gov, openFDA, EMA, KEGG, CDE, publications, and other upstream systems do not replace local source snapshot assets; the materialized asset in this repo is the immutable captured snapshot row with bytes/hash/provenance. (https://docs.dagster.io/guides/build/assets/external-assets)
• Optional source observations can record upstream manifest/version metadata, but downstream assets depend on native sources.* snapshots, not on live URLs or remote APIs.
• When one external request computes multiple durable grains, use distinct asset keys in one Dagster multi_asset computation. CTGov discovery is the reference: run metadata and candidate decisions remain separate assets/tables but share one HTTP traversal and transaction. (https://docs.dagster.io/api/dagster/assets#dagster.multi_asset)
• All source families register artifacts through lib/provenance/source_artifacts.py. Existing keys accept only the same source identity, immutable locator, and exact bytes. Seeing that artifact again in a later capture window is a no-op: the first receipt remains immutable while the Dagster materialization records the newer window. The database trigger prevents changes to the stored artifact key, source identity, release, immutable URI, content hash, or hash algorithm. Definition modules and source-family repositories must not hand-write this shared ledger.
• A sensor observes new reviewed curation decisions and requests targeted rematerialization of affected downstream assets (see 05-agentic-curation-and-overrides.md). (https://docs.dagster.io/guides/automate/sensors)

Partition policy (decided defaults; slices refine)

• Time/release partitions for source snapshots and append-only feeds only when the source exposes a replayable release/window or the asset can re-fetch that window deterministically.
• Source partitions for multi-source layers (e.g. approvals from FDA/EMA/KEGG/CDE) when each source adapter has a distinct source contract.
• Entity-key partitions only where targeted recompute of canonical/q_* is worth the complexity.
• Avoid multi-dimensional partitions unless both dimensions are natural; keep backfills comprehensible. Dagster supports multi-dimensional partitions, but the docs should require a clear natural axis before using them and should respect Dagster’s recommended ceiling of about 100k partitions per asset. (https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets)
• If a source only exposes the current view, do not invent fake historical Dagster partitions. Capture append-only snapshots keyed by source identity, source release where known, fetched time, and content hash.

Partition Matrix from Slice Pressure

Asset family	Default partition	Secondary key / targeting	Reason
source snapshots	pull date/time	source family	source freshness, replay, and parser drift isolation
document artifacts	source document version or seen date	source family + document id	labels, publications, filings, news, registry snapshots need exact provenance
evidence claims	upstream snapshot/document partition	claim kind / subject kind	lets extraction rerun without rewriting canonical state
resolution candidates	subject-key batch	entity type	supports targeted verifier loops and can emit curation review candidates
curation observed assets	decision cursor/watermark	subject key	external agent/human-reviewed store; sensor targets affected subjects
canonical assertions	source/date where natural; otherwise unpartitioned with targeted runs	subject key	avoids over-partitioning high-relationship assets
query read models	unpartitioned stable view; physical backing chosen per family	full-family generation promotion, in-place view replacement, or an explicit incremental contract	publication safety and storage strategy are separate; only clinical trials currently has atomic last-known-good generations
search projections	source row/document partition	embedding model/version	rebuild chunks/vectors independently

Mandatory metadata (decided)

Every materialization should attach: row count, source/extractor/prompt/schema versions where relevant, freshness timestamp, and compact provenance metadata. Dagster/OpenLineage handle how assets/columns transformed; q_provenance handles where/why domain evidence and trust receipts for agents.

Metadata key	Applies to	Meaning
row_count	all materialized table/view assets where measurable	Number of rows produced or refreshed.
source_family	source/normalized/evidence	Upstream source namespace such as ctgov, aact, dailymed, pubmed, sec.
source_snapshot_id	source/normalized/evidence	Pull, release, archive, or document snapshot used.
source_artifact_hash	document artifacts/evidence/provenance	Hash or checksum for exact extraction input.
source_immutable_uri	document artifacts/evidence/provenance	Internal immutable URI for the exact artifact; live URL alone is insufficient.
source_release_or_snapshot	source/provenance	Native source release, archive id, API snapshot id, or document version.
extractor_version	evidence/provenance	Parser, prompt, model, or extraction-schema version.
inference_batch_job_id	batch inference provenance	Durable local batch id when native async batch was used.
curation_decision_id	canonical/query/provenance where an overlay applies	Effective reviewed curation decision affecting the fact.
curation_review_state	curation/canonical/query where an overlay applies	Review state used by the curation_review_state gate.
contract_version	canonical/query	Data contract version used to publish.
check_gate_ids	canonical/query	Blocking/warning gates attached to the asset.
provenance_row_count	query	Number of q_provenance receipts emitted for provenance-required rows/facts.
provenance_bundle_count	query	Number of distinct explanation bundles emitted for provenance_grain=bundle.
freshness_observed_at	source/query	Most recent upstream observation time represented.
publication_protected	query/check	Whether the relation was validated as an isolated candidate before atomic family promotion. true for the clinical-trials family; false for current in-place families.

For inference-producing assets, extractor_version is a roll-up label. Materialization metadata includes the task key, request/outcome counts, accepted/quarantined counts, and batch job id when applicable; exact contract/profile/model/token/cost details live at request/attempt grain in the inference lifecycle rather than being duplicated incompletely on every asset event.

Metadata must be machine-readable enough for tests and operations docs, not just markdown prose.

Check naming convention

Asset checks should be named as <gate_id>__<asset_leaf>, e.g. schema_contract__q_trials or source_authority__approval_indications. The gate id must match 08-contracts-and-quality.md so docs, implementation, and alerts use one vocabulary.

Lineage and provenance strategy

Internal transformation lineage is Dagster-native: asset dependencies, materializations, metadata, and column-lineage metadata show how pipeline assets depend on each other. Where external lineage interchange is needed, align metadata with OpenLineage’s dataset/job/run + facets model rather than inventing a parallel scheme. (https://openlineage.io/docs/spec/facets/)

Domain provenance is separate: source artifacts, extraction runs, evidence claims, resolution candidates/verifier receipts, canonical assertions, reviewed curation overlays, and assertion-provenance links explain where a fact came from and why it is trusted. Those receipts surface through query.q_provenance (see 07-sql-query-layer.md) and should be modelled as first-class assets/tables, not as opaque materialization notes.

Locked Graph Choices

• Model resolution as one asset per entity type. Entity-specific policies and receipts remain visible in lineage instead of being hidden behind one partitioned resolver.
• Curation observation drives owner-routed launches through a sensor. Clinical trials performs blocking candidate validation inside its atomic publication transaction; other families still need an equivalent protected promotion path. A sensor trigger never bypasses checks.