12 Source First Ingestion

Source-First Ingestion

Status: SHARP. Primary sources are the foundation; legacy Data Gov is a read-only cache/reuse layer; paid LLM gap-fill comes last. Denoising state: sharp (noise -> rough -> sharp -> converged) Updated: 2026-07-07

Purpose

Define the clean implementation path for consuming the same external sources that legacy Data Gov consumes, without letting Rails table shapes or legacy llm_data become the new architecture.

The rule is:

primary source bytes first
→ source-compatible snapshots
→ normalized source records
→ deterministic/evidence extraction
→ legacy LLM reuse if the old output matches this source record
→ paid LLM only for remaining gaps

Legacy Data Gov is useful, but only as a read-only historical cache. It must never bypass source contracts, quality gates, or provenance.

Non-negotiables

1. New source contracts own the shape. Legacy tables do not define new schemas.
2. Every source artifact is immutable. Store raw payload bytes/JSON, source-native id/version, artifact hash, fetched/observed timestamps, and immutable URI.
3. Normalized rows are source-faithful. No canonical identity, no LLM output, no curation decisions in normalized tables.
4. Legacy reuse is downstream of normalization. A legacy row is reusable only after it matches a current normalized/source row.
5. Paid LLM is gap-fill only. Fresh provider calls are scoped from migration.legacy_llm_reuse_decisions, validation failures, and true source gaps.
6. No silent trust. Legacy reuse creates evidence claims with legacy locators and import versions, not canonical facts.

Architecture

flowchart TD
  External[External primary sources]
  Source[Source snapshots\nsources_*]
  Normalized[Normalized source records\nnormalized.*]
  Deterministic[Deterministic extractors]
  Legacy[Legacy Data Gov cache\nmigration.legacy_*]
  Reuse[Reuse decisions\nmigration.legacy_llm_reuse_decisions]
  Evidence[Evidence claims\nevidence.*]
  Resolution[Resolution]
  Canonical[Canonical]
  Query[query.q_*]
  LLM[Paid LLM gap-fill]

  External --> Source --> Normalized
  Normalized --> Deterministic --> Evidence
  Legacy --> Reuse
  Normalized --> Reuse
  Reuse -->|reused| Evidence
  Reuse -->|needs_fresh_llm| LLM --> Evidence
  Evidence --> Resolution --> Canonical --> Query

Same source families as legacy Data Gov

Legacy Data Gov docs and services show these upstreams for the major approval/intelligence paths:

Family	Legacy Data Gov source	New repo contract
FDA approvals/products	openFDA download API, especially drug/drugsfda; FDA Drugs@FDA, Orange Book, Purple Book; embedded multi-authority label metadata	sources_regulatory.approvals_snapshots, normalized.approval_source_records, normalized.label_documents
FDA labels	openFDA drug label API and DailyMed SPL API	sources_regulatory.fda_label_* + normalized.regulatory_label_*; exact sections fail closed until governed approved-use decomposition
FDA supplements	FDA REMS CSVs and the FDA companion diagnostic device listing; approval notifications and breakthrough designations are later event/designation grains	sources_regulatory.supplement_snapshots, normalized.regulatory_supplement_records, resolution.approval_supplement_links, then existing safety/CDx evidence
EMA	EMA public XLSX medicine data exports	regulatory snapshots with source_family='ema'
KEGG/PMDA Japan	KEGG drug database / KEGG REST-style API / scraper in legacy	regulatory snapshots with source_family='kegg'
CDE/NMPA China	CDE/NMPA database/portal represented by ChinaApprovedDrug in legacy	regulatory snapshots with source_family='cde'; adapter must preserve Chinese/original text + translation metadata
Clinical trials	ClinicalTrials.gov via AACT, CTGov service, ChiCTR	sources_ctgov, sources_aact, sources_chictr snapshots
Publications	PubMed, Crossref, PMC/Unpaywall, conferences (ASCO/AACR/ASH/EHA/ESMO)	sources_pubmed, sources_crossref, sources_conferences
News/intelligence	Business Wire, GlobeNewswire, Cision, AccessWire, FMP/financial feeds	sources_news.*
Organisations/filings	FMP, SEC submissions/filings	sources_organisations.*, sources_sec.*
Epidemiology	CDC Data Query System initially; explicit profiles for SEER and other official registries	sources_epidemiology.statistic_snapshots
Product commercial facts	SEC EDGAR filing documents and Inline XBRL facts	sources_commercial.filing_document_snapshots
SOC/guidelines	FDA labels + guideline/SOC processing in legacy, NCCN-derived workflows	labels remain regulatory approved-use evidence; professional guideline portals require explicit source/licensing contracts before SOC ingestion

Official-source notes for approvals

FDA / openFDA

Use openFDA bulk downloads for repeatable historical ingestion, not page scraping.

• Manifest: https://api.fda.gov/download.json
• drug/drugsfda partition is present in the manifest as a JSON zip file.
• drug/label is partitioned into multiple JSON zip files.
• openFDA labeling docs say labels are SPL-derived living documents, updated weekly, and converted/annotated to JSON.

Source contract implications:

source_family = fda
source_record_id = stable application/product/document identifier
source_release = openFDA manifest timestamp/hash or download file metadata
source_immutable_uri = openFDA download file URL + partition + record locator
source_artifact_hash = sha256(canonical raw payload)
raw_payload = exact openFDA/drugsfda or label JSON record

DailyMed / SPL

DailyMed provides REST endpoints for SPL metadata and full SPL XML. Legacy uses a DailyMedClient to fetch SPL lists by application number and full SPL XML by set id.

Source contract implications:

source_family = fda_label / dailymed if represented separately, or fda with artifact_kind=label
source_record_id = setid + spl_version
source_release = published_date / fetched_at
raw_payload = exact SPL XML converted to text/json envelope, not lossy parsed fields only
label_documents.document_id = setid
label_documents.label_version = spl_version

EMA

EMA publishes downloadable medicine data tables, including centrally authorised medicines and post-authorisation procedures. EMA says the website updates the data tables overnight.

Source contract implications:

source_family = ema
source_record_id = product number / EMA medicine id / authorization identifier
source_release = XLSX file hash + fetched date
raw_payload = exact row JSON derived from downloaded XLSX + workbook metadata

KEGG

KEGG exposes a REST-style API with operations such as list, get, find, link, and conv; KEGG drug entries and cross-references can be retrieved by KEGG drug id.

Source contract implications:

source_family = kegg
source_record_id = KEGG drug id (e.g. Dxxxxx)
source_release = fetched date + KEGG endpoint URI/hash
raw_payload = exact KEGG flat-file response + parsed row projection

CDE/NMPA

Legacy Data Gov stores Chinese approvals as ChinaApprovedDrug; the direct upstream appears to be a CDE/NMPA database/portal rather than a simple stable public JSON API in the current codebase. The new pipeline must treat this as a first-class source adapter with explicit provenance and translation requirements, not as a legacy-table shortcut.

Source contract implications:

source_family = cde
source_record_id = CDE/NMPA application or drug record id
source_language = zh when original Chinese is captured
translation_version required when English normalized fields are derived
raw_payload = exact portal/API/export row plus original text

Approval ingestion target contract

All approval source adapters must write this stable contract:

sources_regulatory.approvals_snapshots
  snapshot_id uuid primary key
  source_family fda|ema|kegg|cde
  source_record_id text
  source_release text
  source_immutable_uri text
  source_artifact_hash text
  source_language text
  raw_payload jsonb
  observed_at timestamptz
  fetched_at timestamptz

Then existing assets consume the same shape:

normalized.approval_source_records
normalized.label_documents
normalized.regulatory_label_*
evidence.approval_claims
evidence.indication_claims
evidence.regulatory_safety_claims
resolution.approval_keys
resolution.regulatory_entities
canonical.drug_approvals
canonical.approval_indications
query.q_drug_approvals / q_approval_indications / q_label_evidence

Approval source-first implementation

Scope

Implement approvals before other real sources because bounded validation already showed strong legacy reuse:

4 current approval source records evaluated
2 reused legacy output
135 claims reused
135 provider calls avoided
0 provider calls made

Source order

1. FDA application/product databases

   • fetch manifest;
   • download the drug/drugsfda zip;
   • stream records;
   • download the Orange Book zip (products.txt, exclusivity.txt, patent.txt);
   • download the latest Purple Book CSV from the Purple Book downloads page;
   • filter oncology/hematology candidates only after raw artifact capture, not before provenance;
   • write sources_regulatory.approvals_snapshots.

2. FDA supplement artifacts that attach to applications

   • fetch the official REMS CSV exports (csvAllRems, csvRemsProduct, csvMaterials, csvModification);
   • fetch the official FDA companion diagnostic device table;
   • write one raw supplement artifact per REMS program or CDx device row to sources_regulatory.supplement_snapshots;
   • project source-faithful routing rows to normalized.regulatory_supplement_records;
   • link to approval source records only through exact source application numbers in resolution.approval_supplement_links;
   • emit REMS/CDx rows into evidence.regulatory_safety_claims, so downstream q_label_evidence and q_companion_diagnostics materialize through the existing graph.

3. openFDA drug/label or DailyMed SPL labels for matched applications

   • fetch only an exact configured application cohort or an explicitly bounded search;
   • preserve SPL set id/version, full payload, exact source section, and active-version decision;
   • write immutable sources_regulatory.fda_label_* records;
   • project selected documents into normalized.regulatory_label_*;
   • keep compound live indication sections as projection_kind='source_section' until governed approved-use decomposition is implemented and evaluated;
   • never reinterpret a regulatory approved use as standard of care.

4. EMA XLSX

   • download official workbook;
   • hash workbook;
   • convert rows to JSON payloads with workbook metadata;
   • write sources_regulatory.approvals_snapshots with source_family='ema'.

5. KEGG drug entries

   • use KEGG list/get/link as needed;
   • capture raw flat-file response and parsed fields;
   • write sources_regulatory.approvals_snapshots with source_family='kegg'.

6. CDE/NMPA

   • implement only after the source contract is explicit: endpoint/export/manual snapshot, original text, translation policy, source-language metadata.
   • Do not hide CDE behind legacy Data Gov rows.

FDA supplement side spine

REMS and companion diagnostics have their own source grain because they are official regulatory artifacts, not approval rows:

sources_regulatory.supplement_snapshots
  artifact_kind = fda_rems | fda_companion_diagnostic
  raw_payload = exact REMS grouped CSV payload or CDx table row
  source_artifact_hash = sha256(canonical raw payload)

→ normalized.regulatory_supplement_records
  source-faithful application_numbers, drug_terms, disease_terms, biomarker_terms

→ resolution.approval_supplement_links
  accepted only when a source application number exactly matches
  normalized.approval_source_records.application_number

→ evidence.regulatory_safety_claims
  REMS publishes as safety_type = rems
  CDx publishes as safety_type = companion_diagnostic

The FDA supplement live path is selected through regulatory_source Dagster resource configuration, not through asset-body environment branches. Fixtures remain the default for tests and definition loading; live smoke runs may set fda_supplement_mode=live and max_live_records in run config. FDA approval notifications and breakthrough designations are intentionally deferred because they need event/designation evidence grains, not safety/CDx rows.

Explicitly not source-truth shortcuts

• paid LLM gap-fill;
• broad source crawling without source-specific watermarks;
• canonical direct imports;
• old Data Gov as the approval source of truth.
• REMS, companion diagnostics, FDA approval notifications, and breakthrough designations as fake approval rows; REMS/CDx now use the supplement side spine above, while notifications/breakthrough still need event/designation grains before ingestion.

Source-capture contract

Each source adapter should expose typed capture records. For approvals the row-level contract is:

class ApprovalSourceRecord(TypedDict):
    source_family: Literal['fda', 'ema', 'kegg', 'cde']
    source_record_id: str
    source_release: str
    source_immutable_uri: str
    source_artifact_hash: str
    source_language: str
    raw_payload: dict[str, object]
    observed_at: datetime | None
    fetched_at: datetime

Each adapter must be a pure fetch/transform boundary:

HTTP/download bytes -> raw artifact hash -> source snapshot row

No identity resolution, LLM output, or curation in the adapter.

The source-specific Dagster resource is the public interface for asset code. It owns fixture/live mode, paging/bounds, source-specific retry classification, and the low-level source_http dependency. Asset modules call the resource, persist through a repository, and emit metadata; they do not choose fixture vs live by reading environment variables inside the asset body.

defs/sources/<family>/*.py          thin Dagster assets only
lib/sources/<family>/models.py      typed capture records and source-specific value objects
lib/sources/<family>/adapters.py    fixture/live bytes -> typed capture records
lib/sources/<family>/resource.py    ConfigurableResource seam; mode, bounds, HTTP dependency
lib/sources/<family>/repository.py  DDL/upsert/provenance writes

Watermark and partitioning policy

Use source-appropriate watermarks:

Source	Partition	Watermark
openFDA download manifest	manifest fetched date / file hash	manifest file URL + download metadata
Orange Book zip	file hash / fetched date	zip hash + source file + application/product number
Purple Book CSV	release month / CSV hash	CSV URL + row number + BLA/product number
openFDA label	manifest fetched date / label effective time	manifest + label id/set id
DailyMed SPL	setid + spl_version	published_date / spl_version
EMA XLSX	workbook fetched date / workbook hash	workbook hash + updated overnight cadence
KEGG	fetched date + KEGG id	KEGG id + response hash
CDE/NMPA	export/snapshot date	source export hash + original-language row id

Backfills should target source partitions/files, not the whole graph.

Legacy reuse inside approval pipeline

After normalized.approval_source_records exists:

migration.legacy_approval_llm_outputs
+ normalized.approval_source_records
→ evidence/legacy_approval_claims
→ provenance/legacy_llm_reuse_decisions

Reuse states decide spend:

State	Meaning	Provider action
reused	Legacy output matched current source and created claims	no provider call
needs_fresh_llm	No matching legacy output	candidate for future LLM
rejected_schema_mismatch	Legacy row matched but mapper created no valid claims	candidate for future LLM / mapper improvement
rejected_missing_locator	Legacy output lacks enough source locator	do not trust; candidate for LLM
rejected_stale_source	Source artifact changed after legacy output	re-run deterministic/LLM
rejected_low_confidence	Legacy confidence/review state insufficient	candidate for verifier/LLM

No paid call should run unless a cost-cap check sees only needs_fresh_llm/rejected rows and an explicit operator flag enables providers.

Validation gates before live use

Add these checks before declaring approvals live:

1. source_artifact_immutability: same source_family + source_record_id + source_artifact_hash is immutable.
2. approval_source_contract: all adapters populate required snapshot fields.
3. label_version_required: label-derived claims must retain label document id/version or source locator.
4. non_en_translation_required: CDE/non-English rows require original text and translation version.
5. legacy_reuse_locator_required: reused legacy claims must carry legacy table/pk and current source record key.
6. legacy_provider_calls_zero: reuse decision path must keep provider calls at zero.
7. approval_reuse_decision_completeness: every current normalized approval source record has one reuse decision for the approval indication task.
8. approval_gap_budget: fresh LLM candidates must be counted before providers are enabled.

Implementation issues

Issue 1 — FDA application/product source adapter

• Implemented source-capture module: lib/sources/regulatory/ owns typed models, live/fixture adapters, RegulatorySourceResource, and RegulatorySourceRepository; defs/sources/regulatory/approvals.py stays a thin Dagster asset declaration file.
• Fetch https://api.fda.gov/download.json.
• Locate results.drug.drugsfda.partitions.
• Download and stream the zip JSON.
• Download Orange Book from https://www.fda.gov/media/76860/download and project product rows with exclusivity/patent payloads.
• Pick the newest Purple Book CSV from https://purplebooksearch.fda.gov/downloads and project BLA/product rows.
• Write sources_regulatory.approvals_snapshots.
• Default remains fixtures in tests. Live mode is selected by Dagster run config on regulatory_source, e.g. fda_approval_mode=live, with max_live_records only for bounded smoke runs.

Issue 2 — label adapter

• Use openFDA drug/label partitions or DailyMed SPL API.
• Preserve SPL set id/version, label effective/published date, source sections.
• Persist exact versions in sources_regulatory.fda_label_* and project the active selection to normalized.regulatory_label_*.
• Keep normalized.label_documents for embedded multi-authority document identity from approval snapshots; it is not the exact FDA section store.
• Treat live approved-use decomposition and professional-guideline ingestion as separate capabilities.

Issue 3 — EMA XLSX adapter

• Download EMA medicine data workbook.
• Preserve workbook URI/hash/sheet/row metadata.
• Convert each row to raw JSON payload and source snapshot.

Issue 4 — KEGG adapter

• Use KEGG REST list/get/link for drug entries/cross-references.
• Preserve raw flat-file response.

Issue 5 — CDE/NMPA source contract discovery

• Identify the current public source endpoint/export/manual snapshot process.
• Do not implement until original-language and translation provenance are specified.

Issue 6 — approval LLM reuse gate

• Extend migration.legacy_llm_reuse_decisions into a blocking or warning check.
• Every current approval source record must be reused, needs_fresh_llm, or explicitly rejected with a reason before provider execution.

Dagster source-capture design standard

The source-capture module is a deep module: one small source resource interface hides fixture/live selection, paging, raw-source quirks, and retry/error classification. Dagster assets stay thin and legible in the graph.

Source module shape

defs/sources/<family>/*.py          Dagster assets only
lib/sources/<family>/models.py      typed capture records and source value objects
lib/sources/<family>/adapters.py    fixture/live bytes -> typed capture records
lib/sources/<family>/resource.py    ConfigurableResource seam; mode, bounds, HTTP dependency
lib/sources/<family>/repository.py  DDL/upsert/provenance writes
tests/...                          resource, adapter, repository, and asset smoke tests

The deletion test: if lib/sources/<family> were removed, fixture/live branching, source error handling, pagination, hashing, and write semantics would spread back into asset bodies and tests. That means the module is earning its keep. If a helper only passes arguments through, delete it.

Responsibility split

Module	Owns	Must not own
Asset in defs/sources/<family>	Dagster key/group/kinds/owners, dependencies, partition/pool/retry policy, calling source resource, calling repository, emitting materialization metadata	HTTP calls, environment parsing, fixture/live branching, source schema drift handling
Source resource	Dagster ConfigurableResource, fixture/live mode, max_live_records, base URLs, nested source_http, paging policy, retry/error classification	Database writes, canonical decisions, curation, LLM calls
Adapter	Parse source-specific bytes/JSON/CSV/XML into typed capture records with source locators and hashes	Dagster definitions, Postgres transactions, identity resolution
Repository	DDL/upsert/delete policy for source snapshot tables, source-artifact registration, row counts	Network calls, parser branching, source mode

When one source request necessarily computes multiple durable grains, expose those grains as distinct assets but materialize them with one Dagster multi_asset computation and one repository transaction. CTGov discovery is the reference: the run and candidate tables are separately addressable assets, while one paginated request owns both. Do not repeat an external fetch merely to preserve one-function-per-asset aesthetics.

Dagster supports launch-time resource configuration and nested resource dependencies. Use that instead of asset-body environment branches. For secrets or deployment-specific values, prefer Dagster EnvVar; do not call os.getenv while loading definitions because that freezes behavior when the code location starts. (https://docs.dagster.io/guides/build/external-resources/configuring-resources, https://docs.dagster.io/guides/operate/configuration/using-environment-variables-and-secrets)

Derived-demand source capture follows the same rule. Drug and disease/biomarker authority lookups start from evidence terms, but the raw NCIt/RxNorm/MeSH payload is still a source snapshot: the repository chooses pending missing requests, the authority_source resource owns fixture/live and HTTP behavior, and evidence assets parse only after immutable payloads are written.

Configuration rule

Default resources are deterministic and offline. Live mode is selected through Dagster run config, job config, or schedule/sensor-produced run config:

resources:
  ctgov_source:
    config:
      mode: live
      max_live_records: 10
  authority_source:
    config:
      mode: live
      max_live_records: 10
      refresh_policy: missing_only
  target_authority_source:
    config:
      mode: live
      max_live_records: 10
      refresh_policy: missing_only
  epidemiology_source:
    config:
      mode: live
      max_live_records: 10
  sec_source:
    config:
      mode: live
      user_agent: "Bioloupe Data monitored-contact@example.com"
      requests_per_second: 5
      ciks: ["0000310158"]
      commercial_filing_urls:
        - https://www.sec.gov/Archives/edgar/data/.../filing.htm
      max_live_records: 1

Deployment tier is a second, independent safety boundary. BIOLOUPE_RUNTIME_ENV accepts only development, test, staging, or production; a misspelling fails. Every fixture-capable source resource rejects fixture mode in staging and production before fixture bytes are read. The production Compose webserver and daemon set BIOLOUPE_RUNTIME_ENV=production. This guard does not select a live adapter: the job or schedule must still provide the exact live/import resource config. Repository-owned, reviewed static taxonomies are versioned project sources and are not test fixtures.

Use max_live_records for live bounded smoke runs only. It is not a product cutoff, cohort filter, or fixture-size name. Any materialization that uses it must emit:

source_mode = live
capture_scope = bounded_smoke
max_live_records = <N>
is_complete_source_capture = false

Full scheduled/source-refresh runs should leave max_live_records unset and emit capture_scope=complete_source_window for the selected partition/window. Fixture mode may use fixture-specific limits internally, but those should be named fixture_* or kept in the fixture itself, not overloaded onto max_live_records.

Idempotent source writes

Every source/normalized/evidence write is keyed on stable identity, never blind append:

sources_regulatory.approvals_snapshots: unique (source_family, source_record_id, source_artifact_hash)
normalized.approval_source_records: primary key source_record_key
evidence.approval_claims / indication_claims: deterministic claim_id
migration.legacy_llm_reuse_decisions: unique (slice, source_record_key, task, prompt_version, schema_version)

Re-running unchanged input is a no-op. A changed source produces a new source_artifact_hash and therefore a new immutable snapshot row, never a silent overwrite. Downstream current tables choose one deterministic current snapshot per source record; historical snapshots remain append-only in sources_*. Fixture replay follows the same rule: it may select fixture scope in a disposable offline graph, but it never clears live or imported snapshots already present in a development database.

provenance.source_artifacts enforces this contract twice: the shared registration helper accepts only the same source identity, immutable locator, and byte-identical replay for an existing key. An unchanged artifact seen in a later capture window remains one artifact: the original receipt is not rewritten, and the new capture window remains visible in Dagster materialization metadata. A database trigger rejects updates to the stored artifact key, source identity, release, immutable URI, hash, or hash algorithm. Source-family repositories must call the shared helper; Dagster definition modules must never write the ledger directly.

Versions, hashes, partitions, and automation

Do not assign Dagster code_version to live source-capture assets. Their result varies with external state and launch-time resource configuration, so they are not deterministic functions of Dagster asset inputs. Dagster explicitly recommends code_version only when the same inputs deterministically produce the same output. Persist source adapter/parser/rule versions in capture rows and artifact metadata instead; reserve Dagster code_version for deterministic transforms and DDL assets. (https://docs.dagster.io/api/dagster/assets#dagster.asset)

Source snapshots carry source_artifact_hash as the data-version anchor:

• unchanged upstream bytes => unchanged hash => downstream not stale;
• changed bytes => new hash => downstream evidence/reuse recompute is warranted;
• legacy reuse compares its payload hash against the current artifact hash before reuse.

Partition only by natural source release/window when that window can be replayed or targeted. Dagster recommends staying under about 100k partitions per asset; if a source only exposes a current view, capture append-only snapshots instead of inventing fake backfillable partitions. (https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets)

Use declarative cadence (AutomationCondition.on_cron) or an explicit schedule/job when launch-time resource config is required; use sensors for persisted external async state. Blocking checks stop later selected work. Only a query family that validates a frozen candidate before atomic promotion can guarantee that a bad refresh leaves the prior served rows intact. Clinical trials currently has that guarantee; other query families remain in-place and must stay disabled for unattended source refresh until their publication/recovery gate is accepted.

Checks, metadata, concurrency, and retries

Every source/reuse asset attaches checks:

• blocking: source-artifact immutability, required source fields, label/document version, language/translation provenance, locator coverage, zero paid provider calls unless explicitly enabled;
• warning: freshness drift, source coverage gaps, and volume anomalies against the previous source window.

Every materialization emits row counts, source family, source mode, capture scope, source release/window, hash summary, insert/update/no-op counts, provider-call counts where relevant, and code/extractor version.

Use Dagster concurrency pools on live source assets (pool="source_fda", pool="source_ema", pool="postgres") so backfills cannot stampede external APIs or local Postgres. Retry transient HTTP/network failures; fail closed on schema drift, missing source identifiers, impossible dates, bad checksums, or translation/provenance gaps. (https://docs.dagster.io/guides/operate/managing-concurrency/concurrency-pools)

External state and local snapshots

External assets are useful to observe upstream systems that Dagster does not materialize, but Dagster cannot directly materialize or schedule external assets. The materialized asset in this repo is the local immutable snapshot row. Do not treat an external URL, API response promise, or manifest pointer as “captured” until bytes are stored or represented as canonical raw payload, hashed, and linked to a local source-artifact row. (https://docs.dagster.io/guides/build/assets/external-assets)

Live tests are opt-in

Default tests and make check never depend on live external APIs. Each live-capable source family has three tiers:

1. fixture/golden unit tests for parser and contract shape;
2. local DB materialization smoke with checked-in fixture bytes;
3. opt-in live smoke that fetches one bounded window with max_live_records and verifies only source contract, hash, and row-count behavior.

Rejected source-capture patterns

Pattern	Why rejected	Replacement
if BIOLOUPE_LIVE_SOURCE_*: fetch live else: read fixture inside an asset	Puts the seam in the wrong place, freezes behavior at code-location load, and makes tests assert implementation branches instead of source contracts	Source-specific ConfigurableResource with fixture default and run-config live mode
Asset accepts low-level source_http directly for a source family	Forces every asset to understand HTTP and mode details	Source-specific resource depends on source_http internally
max_records as a generic cap	Ambiguous: fixture size, live smoke cap, product sample, or safety guard	max_live_records only for live bounded smoke; fixture limits use fixture names
External assets instead of local source snapshots	Dagster cannot materialize/schedule external assets; agents need immutable local provenance	Optional observation plus native sources.* snapshot assets
Reusable helpers in defs/sources/*_common.py	Dagster definition packages become a utility dumping ground	Move non-definition helpers to lib/sources/<family> or shared lib/sources

Deep-scan cleanup backlog

Area	Classification	Required cleanup
CTGov resource defaults previously read source mode from env in lib/resources	Fixed 2026-07-06	default_resources() now binds fixture-default ctgov_source; live mode and max_live_records are run config fields
Regulatory approvals and supplements	Fixed 2026-07-06	RegulatorySourceResource, adapters, and repository now own fixture/live capture; regulatory assets no longer branch on BIOLOUPE_LIVE_SOURCE_FDA_* or accept direct source_http
Drug/disease authority lookups	Fixed 2026-07-07	AuthoritySourceResource, adapters, and repository now own fixture/live capture; authority assets no longer parse lookup-limit env vars, call NCIt/RxNorm/MeSH directly, or accept direct source_http. Pending requests filter existing snapshots before max_live_records is applied.
SEC organisation capture previously embedded fixture data and fixture/live choice in the asset	Fixed 2026-07-23	SecSourceResource, adapters, and repository now jointly own filer and official filing-document capture; SEC assets are thin declarations.
Epidemiology source capture	Implemented 2026-07-23	EpidemiologySourceResource owns fixture/live selection, official CDC dataset identity, complete paging, and bounded smoke semantics.
Endpoint and adverse-event vocabulary capture	Implemented 2026-07-26	ClinicalVocabularySourceResource owns a versioned endpoint catalog plus exact official NCI CTCAE workbook capture. Immutable artifacts are separate from explicit active-capture pointers; bounded live CTCAE smoke is marked incomplete and cannot replace the prior active catalog.
Target, mechanism, modality, and payload capture	Implemented 2026-07-23	TargetAuthoritySourceResource owns HGNC/ChEMBL fixture/live capture and the versioned project catalog. Full pending drug demand is discovered before the live bound; bounded/missing-only runs merge active scopes and downstream processing reads active artifacts only.

SEC live run config must declare an organisation and monitored contact email in user_agent and must keep requests_per_second <= 10. The adapter defaults to 5 requests/second and applies bounded retry/backoff for 429 and transient 5xx responses. | Publications shared helper under defs/sources/publications_common.py | Fixed 2026-07-23 | Reusable acquisition/parsing now lives under lib/sources/publications; definition modules are thin assets. | | FDA label fixture/live/import branching | Fixed 2026-07-26 | Regulatory-owned RegulatoryLabelSourceResource owns source variation, exact cohort/bounds, HTTP, import, active-version, and scoped activation policy; its Dagster asset lives under defs/sources/regulatory/. | | FMP organisation source locator | Fixed 2026-07-23 | source_url is the upstream FMP endpoint; the internal content-addressed locator is stored separately as immutable_uri. | | Snapshot identity and current-row selection | Fixed 2026-07-23 | Source artifact keys use full payload hashes; unchanged bytes replay across later capture windows; publication and organisation normalizers rank one current correction explicitly instead of depending on database row order. | | Fixture bytes in a deployed source run | Fixed 2026-07-25 | BIOLOUPE_RUNTIME_ENV=staging|production makes every fixture-capable source resource reject fixture mode; production Compose sets the tier explicitly and tests cover every resource. |

Checked-in fixture evidence is classified in lib/sources/fixture_manifest.yaml. A fixture proves deterministic graph behavior only; it is not a live source, current source-shape, completeness, or parity receipt. See ADR-0012.

Updatability acceptance

A source is “perfectly updatable” when:

assets are thin Dagster declarations
fixture/live variation is behind one source resource seam
re-running unchanged upstream is a no-op
a changed source record yields a new immutable snapshot + new artifact hash
adapter/parser/extractor/rule-version drift is persisted and forces targeted replay
automation/sensors + blocking checks drive incremental, gated publication
backfills target natural source partitions/windows where they exist
legacy reuse re-evaluates against current artifact hash
provider_calls stays 0 until explicit gap-fill
live external calls are opt-in in tests and selected by run config
source resources own retries/rate limits; repositories own source writes

Done means

For approvals, the clean implementation is done when:

real FDA/EMA/KEGG/CDE source snapshots exist in sources_regulatory.approvals_snapshots
normalized.approval_source_records contains real source-derived rows
legacy approval reuse decisions cover every normalized approval source record
provider_calls remains 0 until explicit paid gap-fill
q_drug_approvals/q_approval_indications publish only provenance-backed facts
source-inclusive no-provider materialization stays green