Skip to content
Docs Central

0005 Clinical Trials Capability Parity Source First Dag

Clinical Trials Capability Parity Uses a Source-First Dagster DAG

Section titled “Clinical Trials Capability Parity Uses a Source-First Dagster DAG”

Status: accepted

Clinical-trials alignment targets capability parity with legacy Data Gov, not literal migration, minimal cleanup, or full operational parity as the first campaign. The clinical-trials slice is the first scope boundary: preserve every clinically useful legacy capability, but rebuild it through primary source artifacts, evidence claims, verifier/resolution receipts, canonical facts, and q_* SQL models rather than by porting Rails workflows, AACT row ids, Thor tasks, or mutable weekly workflow state.

Considered Options

  • Minimal parity: fix only obvious doc/code drift and missing query names.
  • Literal migration: port the Data Gov workflow order, tables, tasks, and AACT-backed data shape.
  • Operational parity first: recreate schedules, scraper runtime, notifications, and reconciliation before closing semantic gaps.
  • Capability parity first: fill the missing clinical-trial evidence, resolution, canonical, and query capabilities in Dagster dependency order.

Decisions From the Alignment Grill

  • Capability parity is the target: preserve useful legacy clinical-trials capability while rebuilding through source-first Dagster assets.
  • The first campaign scope is the clinical-trials slice, with clean contracts to drug, disease/biomarker, endpoint/adverse-event, organisation, and publication owner spines.
  • Implementation order follows semantic dependencies, not the old weekly workflow: graph hygiene first, then scope/eligibility evidence, verifier-backed resolution, trial references and derived registry projections, and finally operational runtime parity.
  • q_trial_result_groups will be added as a diagnostic/support query model. It should be visible in q_catalog and queryable by agents to explain result-group-to-design-arm linkage, but normal clinical fact questions should prefer q_trial_outcomes, q_trial_safety_events, q_trial_design_arms, and q_trial_interventions.
  • Clinical-trials legacy reuse applies the repo-wide source-matched salvage rule from ADR-0006: old trial LLM/review output may be reused only after it matches a current primary-source-derived normalized trial record, and it may enter only as trial evidence, curation/review input, or coverage/reuse metadata.
  • Trial-specific legacy LLM reuse decisions are a blocking precondition before any fresh trial_eligibility_extract provider call. Each current source-derived trial must have a slice_name = 'trials' decision in migration.legacy_llm_reuse_decisions; fresh eligibility extraction runs only for rows/tasks marked needs_fresh_llm.
  • The trial reuse-decision grain is the current normalized trial plus LLM task, not the old Data Gov row/subgroup. Use source_record_key = normalized.trial_records.trial_key, task_name = 'trial_eligibility_extract', prompt_version = 'legacy-data-gov-trials-llm-import-v1', and the current response_schema_version = 'trial-eligibility-evidence-v2' (v1 remains historical); matched legacy clinical-trial/subgroup rows live in details, legacy_table, legacy_pk, and claim-count metadata rather than defining the decision identity.
  • reuse_state is a provider-call gate, not a semantic completeness grade. For task_name = 'trial_eligibility_extract', reused means matched legacy output satisfies the minimum acceptance contract for skipping the fresh eligibility LLM task: it is source-matched to the current normalized trial, carries usable source locators/hashes, maps to schema-valid evidence.trial_eligibility_claims, is not stale against the current eligibility source artifact, and covers every high-impact eligibility facet detected in the current source text that deterministic extraction did not already cover. Partial legacy eligibility evidence stays in evidence/details, but if any detected required facet remains uncovered, the state remains needs_fresh_llm; do not add a partially_reused state. Legacy disease/design salvage is useful and counted separately, but design-only reuse does not satisfy the eligibility extraction task.

Consequences

Phase 0 cleans graph truth before deeper parity work: remove artificial source dependencies such as ChiCTR depending on CTGov, make any salvage-sidecar back-edges explicit, align docs with implemented/reserved trial assets, and prevent provenance receipts for nonexistent q_* models. AACT and legacy Data Gov remain reconciliation sidecars and product-intent evidence; ClinicalTrials.gov and ChiCTR source artifacts remain the source-authority path for trial facts. Fresh paid LLM work is gap-fill only after deterministic extraction and ADR-0006 source-matched legacy reuse are measured; for clinical-trial eligibility, that measurement is a hard spend-control gate, not merely observability. The decision ledger must stay boring: evidence salvage records useful partial evidence, the reuse decision answers whether to skip a provider call, and later quality/completeness checks judge semantic coverage.