11 Legacy Salvage And Backfills

Legacy Salvage and Backfills

Status: SHARP. Legacy Data Gov is a read-only migration cache used to stage/reuse paid-for evidence and reconcile historical identity rows before any broad LLM re-run. Denoising state: sharp (noise -> rough -> sharp -> converged) Updated: 2026-07-23

Purpose

Define how the Dagster replacement uses the legacy Rails Data Gov database during cutover without turning legacy state into opaque final truth. The goal is to avoid unnecessary provider spend while preserving the new system’s provenance-first contract.

The rule is:

if a legacy reviewed/LLM-derived value maps cleanly and passes gates:
    salvage it as provenance-bearing evidence or curation input
elif deterministic source parsing can produce the value:
    parse it from current source artifacts
else:
    run the new LLM runtime on the remaining gap only

Legacy salvage is therefore a cost-avoidance and continuity layer, not a shortcut around the evidence -> resolution -> canonical -> query graph.

Vocabulary

Term	Meaning in this repo
Source-derived bootstrap	Populate canonical identity spines from real external source artifacts or authority vocabulary snapshots. Legacy Data Gov rows are not source authority for this step.
Legacy reconciliation baseline	A provenance-bearing staging table for legacy identity rows, stable Bioloupe ids, historical source-code mappings, and review context. It may support matching, audits, and continuity reports, but it does not mint canonical identities.
Legacy staging	Materialize selected legacy rows under `migration.*` so they can be matched, audited, and reused without treating legacy state as truth.
Backfill	Run or rerun assets across historical partitions/keys, usually because historical rows are missing or asset logic changed. Dagster backfills are strongest when assets are partitioned by source date, document version, source id, or id range.
Salvage	Use legacy Data Gov as a read-only migration cache to import already-paid-for LLM outputs, reviewed decisions, and curated relationships. Salvage stages raw legacy snapshots, then maps only validated/matched rows into evidence or curation inputs.
Gap fill	New deterministic or LLM extraction over only the rows that salvage cannot safely satisfy.

Dagster operating model

Dagster treats assets as durable nouns and backfills as targeted materialization of missing/stale partitions. For legacy salvage this means:

Legacy Data Gov is a read-only migration input. Dagster observes and reads it but never mutates it, and never treats it as source authority.
The import target is materializable. Tables under migration.* are produced by Dagster assets and are safe to truncate/rebuild in local/dev if needed.
Mapped evidence is materializable. Assets such as evidence/legacy_publication_claims read migration rows plus current normalized source artifacts, then insert into the same evidence claim tables used by fresh deterministic/LLM extraction.
Broad LLM backfills come last. lib.llm.unseen_requests(...) and the LLM batch ledger should see only rows that remain missing, stale, invalid, or low-confidence after salvage.

Layering contract

flowchart TD
  Legacy[(Legacy Data Gov RDS\nread-only migration cache)]
  Raw[migration.legacy_*_outputs\nraw selected legacy JSON + locator]
  Evidence[evidence.*_claims\nextractor_kind = llm\nlegacy_data_gov locator]
  Resolution[resolution.*]
  Canonical[canonical.*]
  Query[query.q_*]
  LLM[LLM gap-fill\nonly for uncovered/stale rows]

  Legacy --> Raw --> Evidence --> Resolution --> Canonical --> Query
  Evidence -. uncovered gaps .-> LLM --> Evidence

Do not write legacy values directly into query.q_*. Legacy rows become evidence inputs, reconciliation inputs, or curation/review inputs. Identity spines must populate canonical tables from source artifacts, authority vocabulary snapshots, verified resolution, or reviewed curation decisions, not from old Rails canonical tables.

Provenance requirements

Every salvaged row must retain:

legacy table name and primary key;
source-family locator where available (NCT, PMID, DOI, release id, application number, legacy source_type/source_id);
hash of the imported JSON payload;
import extractor version;
a technical locator marker (source_family = legacy_data_gov) that identifies the migration cache, not source authority;
current source artifact anchor when the legacy row can be matched to normalized.*.

Mapped claims use extractor_kind = 'llm' because the legacy value was LLM-derived, but extractor_version must be a legacy-import version rather than a live provider prompt version.

Safety policy

Legacy salvage assets must be safe by default:

no provider API keys are required;
no paid LLM calls are made;
no legacy/prod writes are made;
legacy_data_gov_source defaults disabled, so a URL present in .env cannot trigger reads;
the URL is supplied explicitly through Dagster run config, normally using an BIOLOUPE_DATALAKE_URL environment-secret reference;
every legacy connection sets default_transaction_read_only=on in addition to using scoped credentials;
max_import_records defaults to 1000; a positive integer is a bound and explicit null requests a complete import.

Example run-config fragment:

resources:
  legacy_data_gov_source:
    config:
      enabled: true
      url:
        env: BIOLOUPE_DATALAKE_URL
      max_import_records: 1000

Where a slice has current normalized source records, the importer prioritizes rows matching those source identifiers before filling the bounded sample window. Fixture/live source coverage can therefore be measured without turning a small sample into an arbitrary first-N import.

Implemented salvage slices

Legacy sampling showed publications, trials, and approvals carry high-value structured outputs that are expensive to recompute:

Legacy cache row	Useful shape
`public.publications.llm_data.registry_records` / `api_registry_records` / `web_search_registry_records`	publication -> trial link claims
`public.publications.llm_data.intervention_arms[].interventions[]`	intervention/treatment mention claims
`public.publications.llm_data.subgroup_outcome_measures[].outcome_measures[]`	result claims with endpoint/subgroup/arm/statistic context
`public.publications.llm_data.adverse_events[*]`	safety/adverse-event claims
`public.clinical_trials.llm_data.disease_extractor`	trial condition/design evidence claims
`public.trial_subgroups.llm_data.treatment_lines` / `prior_therapies`	trial eligibility/treatment-line evidence claims
`public.drug_approvals.llm_data.structured.indications[*]`	approval + indication evidence claims
legacy identity tables (`drugs`, `diseases`, `targets`, `biomarkers`, `organisations`, `clinical_trials`)	reconciliation indexes for matching old ids/names to source-derived canonical identities; never source authority by themselves

Implemented assets:

Asset	Table(s)	Role
`sources/legacy_data_gov/identity_bridge`	`migration.legacy_identity_bridge`	Bounded read-only copy of legacy ids, Bioloupe ids, source-native ids, and names for later identity reconciliation/resolution.
`sources/legacy_data_gov/publication_llm_outputs`	`migration.legacy_publication_llm_outputs`	Bounded read-only copy of legacy publication LLM JSON and source locators.
`sources/legacy_data_gov/trial_llm_outputs`	`migration.legacy_trial_llm_outputs`	Bounded read-only copy of legacy clinical-trial and trial-subgroup LLM JSON.
`sources/legacy_data_gov/approval_llm_outputs`	`migration.legacy_approval_llm_outputs`	Bounded read-only copy of legacy drug-approval LLM JSON.
`evidence/legacy_publication_claims`	existing `evidence.publication_*_claims` tables	Maps matched legacy publication JSON into provenance-bearing evidence claims.
`evidence/legacy_trial_claims`	existing `evidence.trial_*_claims` tables	Maps matched legacy trial/subgroup JSON into provenance-bearing evidence claims.
`evidence/legacy_approval_claims`	existing `evidence.approval_claims` / `evidence.indication_claims` tables	Maps matched legacy approval JSON into provenance-bearing evidence claims.
`sources/legacy_data_gov/news_llm_outputs`	`migration.legacy_news_llm_outputs`	Bounded read-only copy of legacy news LLM JSON.
`sources/legacy_data_gov/soc_llm_outputs`	`migration.legacy_soc_llm_outputs`	Bounded read-only copy of legacy guideline/SOC LLM JSON.
`evidence/legacy_news_claims`	existing `evidence.news_entity_mention_claims` table	Maps matched legacy news entity arrays into provenance-bearing mention claims.
`evidence/legacy_soc_claims`	existing `evidence.soc_treatment_approach_claim` table with `claim_origin='legacy_reconciliation'`	Maps matched legacy output into provenance-bearing reconciliation evidence; it is excluded from approved-use review and canonical SOC publication.

Only legacy rows that match current normalized records are published as evidence. Unmatched legacy rows remain in migration.legacy_* staging tables for later source-derived backfills, identity reconciliation, or fresh LLM gap-fill.

Implemented first-stage asset: sources/legacy_data_gov/drug_baseline writes migration.legacy_drug_baseline, a focused copy of legacy public.drugs plus joined source-code fields and staged ownership context (drug_type, development_phase, therapeutic_areas). This table is reconciliation input only: match old B... ids and historical NCIt/RxNorm/Adis/ChEMBL codes to source-derived identities, preserve ownership-suggestion context for docs/26-slice-organisation-intelligence.md, and report coverage gaps. It must not write canonical.drugs, canonical.drug_aliases, or canonical.drug_external_ids.

Backfill expansion plan

Publication LLM salvage — implemented; proves raw legacy snapshot -> mapped evidence -> normal query path.
Clinical trial disease/eligibility salvage — implemented for clinical_trials.llm_data disease extraction and trial_subgroups.llm_data treatment-line/prior-therapy extraction.
Approval label-indication salvage — implemented for drug_approvals.llm_data.structured approval/indication claims.
News entity/event/deal salvage — implemented for news.llm_data entity mentions; future deepening can add news_drug_mentions, news_trial_mentions, and event/deal claims.
SOC/guideline salvage — implemented as an optional reconciliation sidecar for guidelines.llm_data.old_chemo_search; explicit claim origin prevents it from masquerading as current regulatory approved use or publishing into SOC. Future deepening may compare label trials/approaches but must keep the sidecar boundary.
Drug identity reconciliation — keep public.drugs and public.brand_drugs in migration.* only. Source-derived identity comes from NCIt/RxNorm snapshots, regulatory source records, intervention mentions, verifier receipts, and reviewed curation. Legacy drug/brand rows can attach old ids, aliases, and stale mappings only after they resolve to a source-derived canonical drug; unmatched rows remain staged for review or future source backfill.
LLM gap-fill backfills — only after salvage coverage is measured and gates identify the remaining uncovered rows.

Coverage and reuse ledgers

Implemented assets:

Asset	Table	Role
`provenance/legacy_salvage_coverage`	`migration.legacy_salvage_coverage`	One row per salvage slice, measuring imported rows, matched current sources, evidence claims created, rejected/unmatched rows, remaining gap count, estimated provider requests avoided, and provider calls.
`provenance/legacy_llm_reuse_decisions`	`migration.legacy_llm_reuse_decisions`	Source-first per-record decision ledger for LLM tasks. Implemented for approvals and clinical-trial eligibility: each current source-derived record/task is marked `not_required`, `reused`, `needs_fresh_llm`, or a `rejected_*` state before any provider call is allowed.

For the implemented publication/trial/approval/news/SOC slices, the ledger computes:

legacy_rows_imported from the corresponding migration.legacy_*_llm_outputs table;
matched_to_current_source by source identifiers against current normalized records;
claims_created by counting legacy-located rows in existing evidence claim tables;
claims_rejected as imported legacy rows that did not match current normalized/source records;
remaining_gap_count as current normalized rows not covered by imported legacy rows;
provider_calls = 0 by construction.

Coverage keys by slice:

Slice	Match basis	Evidence asset
Publications	PMID/source id or DOI -> `normalized.publication_records`	`evidence/legacy_publication_claims`
Trials	NCT/ChiCTR/source registry id -> `normalized.trial_records`	`evidence/legacy_trial_claims`
Approvals	source id, application number, or product/generic -> `normalized.approval_source_records`	`evidence/legacy_approval_claims`
News	release id -> `normalized.news_articles`	`evidence/legacy_news_claims`
SOC	disease name -> `normalized.regulatory_label_indication`	`evidence/legacy_soc_claims`

These ledgers are the spend-control surface: fresh LLM gap-fill is scoped from remaining_gap_count and legacy_llm_reuse_decisions.reuse_state, not from total historical source rows. For clinical trials, ADR-0005 makes the trials branch of legacy_llm_reuse_decisions a hard precondition before any trial_eligibility_extract provider call: only needs_fresh_llm rows/tasks are eligible for new work.

For approvals, legacy_llm_reuse_decisions records:

current source_record_key and source_artifact_key;
current source match keys (source_record_id, application_number, product/generic, region);
matched legacy table/pk/hash when present;
match basis (source_id, application_number, product_generic, or none);
task/schema version (approval_indication_decomposition, legacy import prompt/schema versions);
reuse state (reused, needs_fresh_llm, or rejected_*);
created claim count and provider calls avoided;
provider_calls = 0 by construction.

For trials, legacy_llm_reuse_decisions records one decision per current normalized trial and LLM task, not one decision per legacy clinical-trial/subgroup row:

slice_name = 'trials';
source_record_key = normalized.trial_records.trial_key;
task_name = 'trial_eligibility_extract';
prompt_version = 'legacy-data-gov-trials-llm-import-v1';
response_schema_version = 'trial-eligibility-evidence-v2';
matched legacy public.clinical_trials / public.trial_subgroups rows, legacy hashes, match bases, design-claim counts, eligibility-claim counts, and rejected-row counts in metadata/details;
reuse state (not_required, reused, needs_fresh_llm, or rejected_*) describing whether this current source-derived trial still needs the fresh eligibility LLM task.

For task_name = 'trial_eligibility_extract', reuse_state is a provider-call gate, not a semantic completeness grade. reused means the matched legacy output satisfies the minimum acceptance contract for skipping the fresh eligibility LLM task. Partial legacy eligibility evidence is still preserved in evidence tables and details (partial_legacy_reuse, modifiers present, missing/uncovered facets, source-text availability), but if a fresh LLM call is still needed the state remains needs_fresh_llm. Do not add a partially_reused state: partialness belongs in metadata/details, not in the spend-control state machine. Design/disease salvage from clinical_trials.llm_data remains valuable evidence and coverage metadata, but by itself it does not avoid the eligibility LLM task.

evidence/trial_eligibility_gap_fill consumes only the needs_fresh_llm cohort. It is disabled by default, requires an explicit bounded run (max_requests, optionally trial_keys), and records an immutable projection receipt after an accepted result, including accepted zero-claim outputs. not_required means there is no current eligibility text or deterministic evidence already covers every detected high-impact facet; it is not a model judgment.

Minimum acceptance contract for skipping trial_eligibility_extract:

Current source anchor: the decision is for a current normalized.trial_records.trial_key, with current source_artifact_key, registry id, and source-located eligibility text or an explicit no-text reason.
Source-matched legacy input: matched legacy public.clinical_trials / public.trial_subgroups rows match the current source-scoped registry id and carry legacy table, primary key, payload hash, and source locator metadata.
Schema-valid eligibility evidence: accepted legacy outputs create one or more task-relevant evidence.trial_eligibility_claims that pass the current modifier/operator/polarity vocabulary, preserve evidence_text or a source locator, and do not rely on unsupported or malformed legacy JSON shapes.
Freshness: every salvaged legacy eligibility quote is an exact substring of the current source eligibility text. Missing or stale quote support leaves uncovered facets eligible for needs_fresh_llm; legacy evidence is retained but cannot suppress current-source work.
Facet coverage: every high-impact eligibility facet detected in the current source text is covered either by deterministic extraction or by accepted legacy eligibility evidence. Facets include disease/subtype/extent/stage/status/risk/treatment setting, treatment line/prior therapy, biomarker, ECOG, and age. Any detected facet not covered by deterministic extraction or accepted legacy evidence keeps the decision at needs_fresh_llm.

This means a legacy treatment-line/prior-therapy claim can be reused to avoid the fresh task only when the current source text has no other uncovered high-impact eligibility facets. Otherwise the claim is preserved as partial evidence and the provider-call decision stays needs_fresh_llm.

Acceptance gates

A salvage slice is not complete until it can report:

imported raw legacy row count;
matched-to-current-source count;
mapped evidence claim counts by claim table;
skipped count with reason (unmatched_publication, missing_required_span, unsupported_shape, invalid_enum);
zero paid provider calls;
make check passes;
source-inclusive safe materialization fails only on intentionally excluded paid LLM/gap-fill assets.