21 Slice Clinical Trials

Pipeline Slice: Clinical Trials

Status: CONVERGED NCT CORE / ATOMIC QUERY FAMILY / ChiCTR FIXTURE PROFILE. The official NCT source spine, protocol/result grains, governed eligibility extraction, provenance, and generation-protected seventeen-model q_trials family are implemented and fixture-tested. ChiCTR remains source-scoped and fixture-backed; live scraper hardening is an explicit source expansion. The optional eligibility task is provider-activation gated. Denoising state: converged (noise -> rough -> sharp -> converged) Updated: 2026-07-25

Purpose

Model the clinical-trials pipeline end-to-end at the same standard as every other domain slice. It must define its own source snapshots, normalized records, evidence claims, entity resolution, LLM-assisted resolution/verification, curation overrides, canonical assets, SQL read models, checks, operations, and tests without becoming the template for other slices.

Clinical trials are the hardest useful first slice because legacy Data Gov already treated them as a weekly, multi-source, LLM-heavy pipeline: ClinicalTrialsWorkflow collected/reconciled trials, extracted study plans/outcomes/conditions/eligibility, and then triggered ClinicalTrialEligibilitiesWorkflow for disease/biomarker resolution (../bioloupe-data-gov/docs/clinical-trials.md, ../bioloupe-data-gov/app/workflows/clinical_trials_workflow.rb, ../bioloupe-data-gov/app/workflows/clinical_trial_eligibilities_workflow.rb).

Current Decided Shape

Clinical trials is one equal-priority domain slice. It must exercise its domain model, entity resolution, LLM verification, curation overrides, asset graph, contracts, operations, tests, and q_* query layer, while resolving shared identities through owner spines such as 29-slice-drug-identity.md.

flowchart LR
  CTGov[ClinicalTrials.gov API]
  ChiCTR[ChiCTR]
  Raw[Trial source snapshots]
  Normalized[Normalized trial records]
  Evidence[Trial design / intervention / endpoint / eligibility claims]
  Resolution[Drug, disease, biomarker, organisation resolution]
  Verify[Verifier / accepted basis]
  Curation[Reviewed curation overlay]
  Canonical[Canonical trial assets]
  Query["q_trials family"]

  CTGov --> Raw
  ChiCTR --> Raw
  Raw --> Normalized --> Evidence --> Resolution --> Verify --> Canonical --> Query
  Curation -. optional override .-> Canonical

Source Inventory

Source	Greenfield role	Source identifier	Legacy evidence	Web/reference evidence
ClinicalTrials.gov API v2	authoritative NCT registry source for study protocol, status, contacts, interventions, arms, outcomes, eligibility, references, result groups, outcome analyses, baseline characteristics, participant flow, and adverse-event results	NCT{8 digits}	Data Gov used CTGov for version/change reconciliation and AACT as a convenience mirror (../bioloupe-data-gov/docs/clinical-trials.md, clinical_trials:versions:update)	ClinicalTrials.gov publishes a modern REST API, OpenAPI spec, /version, paginated /studies, /studies/{nctId}, and searchable field metadata: https://clinicaltrials.gov/data-api/api and https://clinicaltrials.gov/api/oas/v2
AACT	optional reconciliation/audit mirror for NCT data; not a greenfield source spine and not required for canonical/query facts	nct_id plus mirror-local table ids/group codes	legacy connects read-only to AACT for discovery/results because AACT exposed convenient SQL tables (../bioloupe-data-gov/docs/clinical-trials.md, lib/tasks/clinical_trials/outcomes.thor, ../bioloupe-ai-solutions/data-collection/src/clinical_trials/collect_results_full.js)	AACT is a PostgreSQL mirror of public ClinicalTrials.gov protocol/results data, loaded from ClinicalTrials.gov and roughly one day behind; it does not include study change history: https://aact.ctti-clinicaltrials.org/points_to_consider
ChiCTR	additional registry source for Chinese trials not covered by ClinicalTrials.gov; operationally high-risk due no API / anti-bot	ChiCTR{number}	legacy scrapes weekly with Ferrum/headless Chrome and imports XML/markdown (../bioloupe-data-gov/docs/chictr.md, lib/tasks/clinical_trials/chictr_weekly.thor)	ChiCTR public search/registration site: https://www.chictr.org.cn/
PubMed references	trial-publication linkage seed; detailed publication evidence belongs in 23-slice-publications.md	PMID / DOI / registry ids in abstracts	clinical_trials:references:update_for_new_trials and publication_clinical_trials	PubMed/NCBI E-utilities and MeSH are the publication/search vocabulary family: https://www.ncbi.nlm.nih.gov/books/NBK25501/
NCI Thesaurus / EVS	controlled-vocabulary anchor for oncology diseases, biomarkers, and non-drug concepts	NCIt code	legacy disease/condition/biomarker matching calls NCI services and stores nci_code (../bioloupe-data-gov/app/models/clinical_trial_condition.rb, docs/data-model.md)	NCI EVS provides cancer vocabulary resources including NCI Thesaurus and NCI Metathesaurus: https://www.cancer.gov/about-nci/organization/cbiit/vocabulary
CDISC SDTM	reference vocabulary/grain guide, not a target schema clone	SDTM domains/controlled terminology	legacy has no CDISC-native model; it independently models arms, endpoints, outcomes, and eligibility	CDISC SDTM defines the general conceptual model for representing clinical study tabulation data; use as alignment reference only: https://www.cdisc.org/standards/foundational/sdtm and https://www.cdisc.org/standards/foundational/sdtm/sdtm-v2-0

Source decision

Use a ClinicalTrials.gov API-first NCT spine:

• ClinicalTrials.gov API v2 is the immutable official source-artifact spine for NCT discovery, study JSON, protocol modules, eligibility text, result modules, adverse-event sections, references, API version, source data timestamp, and exact per-study snapshots.
• AACT is not a first-class Dagster source for canonical/query facts. It is retained only as an optional reconciliation/audit mirror because it mirrors ClinicalTrials.gov current public data and exists mainly to make SQL analytics easier. AACT ids never become greenfield identity, and greenfield q_* facts must not depend on AACT row ids.
• ChiCTR is a first-class separate registry source family. ChiCTR can lag operationally, but the canonical model must not assume every trial has an NCT id.
• Historical change backfill is a separate source decision. The CTGov API/OpenAPI surface used here exposes the most recent public study record; our source ledger preserves forward history from first capture. If pre-existing historical versions matter, model the ClinicalTrials.gov archive as its own source family. Do not pretend AACT solves that: AACT also excludes study change history.

Legacy correction: the original bioloupe-ai-solutions production query filtered AACT conditions, browse_conditions, interventions, and studies for oncology/hematology therapeutic trials (../bioloupe-ai-solutions/data-collection/helpers/queries_clinical_trials.js, ../bioloupe-ai-solutions/data-collection/src/clinical_trials/collect_trials.js). That proves the product intent: cohort discovery with source-visible reasons. It does not prove AACT should remain a greenfield source spine now that ClinicalTrials.gov API v2 exposes current protocol/results JSON and searchable condition/intervention areas directly.

Legacy Data Gov / Bioloupe AI source map

The old system is split across two repos. Data Gov orchestrates and enriches the workflow; Bioloupe AI performs the AACT bulk pulls. Greenfield Dagster should preserve the source semantics and product intent, not port the Rails/Thor/Sidekiq/shell choreography.

Legacy capability	Where the old system does it	What the old step really means	Greenfield Dagster model
Weekly clinical-trial orchestration	../bioloupe-data-gov/app/workflows/clinical_trials_workflow.rb	Order AACT, ChiCTR, CTGov history, result, study-plan, condition, eligibility, reference, and notification steps.	Asset dependencies, partitions, checks, and sensors. No workflow class or shell-step clone.
AACT cohort discovery and trial bulk load	Data Gov calls build-clinical-trials.sh; implementation is ../bioloupe-ai-solutions/data-collection/src/clinical_trials/collect_trials.js and helpers/queries_clinical_trials.js	Select interventional oncology/hematology therapeutic trials using study type, intervention type, raw condition text, browse/MeSH condition terms, curated disease terms, cancer keywords, and therapeutic-area terms.	sources.ctgov.trial_discovery_runs and sources.ctgov.trial_discovery_candidates; CTGov query params, returned NCT ids, source search areas, matched rules, accepted/rejected scope decisions, and page tokens are replayable evidence.
AACT eligibility text load	../bioloupe-ai-solutions/data-collection/src/clinical_trials/collect_eligibilities.js	Pull source eligibility text and split inclusion/exclusion sections.	normalized.trial_eligibility_texts from CTGov protocolSection.eligibilityModule.eligibilityCriteria; later evidence.trial_eligibility_claims extracts structured facts.
ClinicalTrials.gov version / change detection	../bioloupe-data-gov/lib/tasks/clinical_trials/reconciliation.thor, eligibility_changes.thor, CtGovService	Compare current registry/source modules, flag meaningful changes, and decide which downstream families need refresh.	sources.ctgov.trial_snapshots stores full payload bytes/hash plus API version/data timestamp; ops.ctgov_snapshot_diffs hashes source modules and targets rematerialization without mutable workflow flags.
AACT stable metadata selective sync	../bioloupe-ai-solutions/data-collection/src/clinical_trials/collect_stable_tables.js using change_flags_config.js	Refresh sponsors, locations, officials, study designs, and documents only when the corresponding CTGov module changed.	Parse CTGov modules into stable normalized families; module-hash diffs decide whether sponsor/location/design/reference downstream assets need refresh.
AACT results sync	../bioloupe-ai-solutions/data-collection/src/clinical_trials/collect_results_full.js and SELECTIVE_SYNC_DOCUMENTATION.md	When any result module changes, replace the full per-trial result hierarchy because AACT result row ids are unstable.	Parse CTGov resultsSection as a full result-family snapshot per trial/source hash. Result-group ids such as OG000 / EG000 are source locators inside the CTGov snapshot, not canonical identity.
ChiCTR collection	../bioloupe-data-gov/lib/tasks/clinical_trials/chictr_weekly.thor, app/services/chictr/*	Scrape/import ChiCTR XML/markdown with run metadata and anti-bot/parser health risk.	sources.chictr.trial_xml_artifacts, sources.chictr.trial_snapshots, and parser-health checks; ChiCTR remains source-scoped identity.
Study-plan extraction and component matching	../bioloupe-data-gov/app/services/study_plan/*	LLM extracts arms/components; matching resolves interventions to drugs/non-drug concepts with exact/manual/context/fuzzy bases.	evidence.trial_design_claims and resolution.trial_entities for trial_intervention_link and trial_result_group_arm_link; curation handles disputed high-impact links.
Eligibility enrichment and disease/biomarker matching	../bioloupe-data-gov/app/workflows/clinical_trial_eligibilities_workflow.rb, ParticipationCriteriaService, TrialEligibilityCriterion	LLM structures criteria, then disease/biomarker candidates are searched, judged, post-processed, and QA’d.	evidence.trial_eligibility_claims with provider-neutral LLM task, schema validation, quarantine, verifier receipts, and resolution through disease/biomarker spines.
ORR / derived result projections	../bioloupe-data-gov/lib/tasks/clinical_trials/derived_measurements.thor	Compute explicit ORR or guarded CR+PR projections without overwriting source results.	Publish source-reported ORR. Do not infer ORR by summing response categories unless a future, separately versioned derivation contract proves compatible denominator, arm, population, and timepoint grain.
API controllers and analytics views	../bioloupe-data-gov/app/controllers/api/clinical_trials_controller.rb, analytics bridge/dim/fact tables	Product questions over trials, diseases, drugs, organisations, eligibility, and results.	SQL-only q_trials family plus q_provenance; no REST/GraphQL consumer surface.

Implementation alignment checkpoint:

• Current source assets now model the NCT spine from ClinicalTrials.gov API v2 shape: fixture mode uses protocolSection, resultsSection, derivedSection, and hasResults; live mode is opt-in through ctgov_source Dagster resource run config and captures /version, paginated /studies discovery, and /studies/{nctId} snapshots.
• sources.aact.result_snapshots is removed from the required Dagster graph. AACT remains only an optional future reconciliation mirror and must not feed canonical/query facts.
• Discovery now materializes accepted and rejected candidates in sources.ctgov.trial_discovery_candidates; q_trial_discovery_candidates publishes both accepted and rejected NCT ids with source-search provenance.
• CTGov protocol/result modules now feed canonical and query models for sites, references, outcome analyses, participant flow, baseline characteristics, adverse events, and adverse-event totals directly from the immutable source snapshot.
• Eligibility extraction now combines a conservative deterministic parser with an opt-in, bounded trial_eligibility_extract gap-fill. Exact artifact/quote checks, requested-facet scope, numeric-bound validation, rejection quarantine, immutable inference decisions, and projection receipts are implemented; this is not presented as an independent semantic verifier.
• The correct source authority is ClinicalTrials.gov API v2 and ChiCTR raw artifacts. Legacy Data Gov rows and AACT mirror rows are reconciliation sidecars and product-intent evidence, not source truth.

Implemented capabilities and explicit boundaries

This table distinguishes executable behavior from deliberate non-behavior. A future extension must earn a producer, contract, tests, and provenance path; this spec does not reserve placeholder assets.

Capability	Status	Note
CTGov NCT source spine (trial_discovery_runs/_candidates/trial_snapshots, ops.ctgov_snapshot_diffs)	implemented	fixture default; live opt-in through ctgov_source run config.
ChiCTR source family (sources.chictr.trial_snapshots)	implemented (fixture)	Independent of CTGov. Live scraper/runtime hardening remains a source-breadth expansion, not hidden NCT fallback behavior.
Normalized/evidence/canonical/q_trials family (17 models incl. support grains)	implemented	Protocol, eligibility, status history, sites, sponsors, references, results, safety, flow, and baseline each retain their own grain. q_trial_result_groups is diagnostic/support (ADR-0005).
Eligibility extraction	implemented	Deterministic claims run first. Optional trial_eligibility_gap_fill reads only needs_fresh_llm rows, defaults disabled, validates exact citations/request scope/numeric bounds, quarantines rejected candidates, and projects accepted current-source evidence with immutable receipts (docs/06,11).
Trial entity resolution	implemented fail-closed	Exact/cross-reference bases may publish. Fuzzy candidates park because the live asset deliberately has no semantic verifier profile yet; adding one requires an evaluated verifier contract, not a default model call.
q_trial_references / canonical.trial_references	implemented	Exposes the registry-stated PMID/citation. Publication identity and accepted publication-to-trial links remain owned by q_publications / q_publication_trial_links (docs/23).
Trial discovery scope	implemented at source grain	sources.ctgov.trial_discovery_candidates is the single owner of accepted/rejected candidate decisions and their query/rule provenance. There is no duplicate evidence/canonical scope assertion.
Derived ORR	intentionally excluded	q_trial_outcomes publishes registry-reported results only. CR+PR is not assumed to be ORR across possibly incompatible groups, populations, denominators, or timepoints.
Cross-registry merging	intentionally unmerged	NCT and ChiCTR remain source-scoped identities. Any future merge requires explicit cross-reference or reviewed high-impact curation evidence.
Legacy trial salvage (sources.legacy_data_gov.trial_llm_outputs -> evidence.legacy_trial_claims -> provenance.legacy_llm_reuse_decisions)	implemented	source-matched legacy evidence salvage (ADR-0006); the reuse-decision ledger is a hard precondition before fresh trial_eligibility_extract provider calls, keyed by current normalized trial + LLM task, and can skip the provider only when the docs/11 minimum source/freshness/schema/facet-coverage contract passes (ADR-0005).

Reported-event sourcing decision

reported_events and reported_event_totals are live clinical-trial result facts, not endpoint-spine vocabulary. Legacy Data Gov sourced them through the AACT mirror: build-clinical-trials-results-selective.sh runs collect_results_full, which fetches reported_events and reported_event_totals from AACT, maps nct_id to clinical_trials.id, maps AACT result_group_id to Data Gov result_groups.id, inserts the two reported-event tables in dependency order, and resets the adverse_events result-change flag only after successful insert (../bioloupe-ai-solutions/data-collection/build-clinical-trials-results-selective.sh, src/clinical_trials/collect_reported_events.js, src/clinical_trials/collect_results_full.js, src/clinical_trials/change_flags_config.js). The older full script had an explicit “Ingesting reported events” step (../bioloupe-ai-solutions/data-collection/build-clinical-trials-results.sh).

Live read-only Data Gov sampling (2026-06) confirms this is not dead data: catalog estimates show reported_events around 5.9M rows and reported_event_totals around 126k rows, while _stage twins are empty outside an active sync. Data Gov light DB dumps exclude reported_events, versions, and embeddables, so a zero-row light snapshot is not liveness evidence (../bioloupe-data-gov/lib/tasks/util/db_sync.thor).

Greenfield model:

• CTGov resultsSection.adverseEventsModule.seriousEvents and otherEvents become trial-owned safety-observation fact families, not adverse_event_terms identity tables.
• CTGov resultsSection.adverseEventsModule.eventGroups totals (deathsNumAffected, seriousNumAffected, otherNumAffected, and matching at-risk fields) become group/event-type denominator/total families, not duplicate event rows.
• AACT ids are never source authority. If AACT is used for reconciliation, its ids remain mirror-local locators inside a reconciliation sample; Data Gov’s selective-sync docs record that AACT result ids are unstable and results tables had to be replaced per trial.
• Adverse-event terms resolve through 32-slice-endpoint-adverse-event-identity.md; the raw source term, organ system, assessment/vocabulary, event type, and counts stay on the trial safety fact.

Cohort filtering / trial discovery contract

We keep the filtering. Without it, the system either ingests too much irrelevant ClinicalTrials.gov material or loses the old product’s oncology/hematology focus. The difference in greenfield is that filtering becomes a first-class, replayable asset with provenance rather than hidden script logic.

flowchart LR
  CTGovSearch[ClinicalTrials.gov search API]
  CTGovStudy[ClinicalTrials.gov study API]
  DiseaseLexicon[Canonical disease + seed cancer keywords]
  SearchAreas[Condition / intervention search areas]
  CandidateFilter[trial_discovery_candidates]
  ApiSnapshots[CTGov study snapshots]
  Evidence[trial_scope_evidence]
  Canonical[canonical.trials]

  DiseaseLexicon --> CandidateFilter
  SearchAreas --> CandidateFilter
  CTGovSearch --> CandidateFilter --> CTGovStudy --> ApiSnapshots --> Evidence --> Canonical

Filtering inputs:

Input	Source	Use
Disease keyword patterns	curated disease names/synonyms + seed oncology/hematology keyword list	Catch free-text Condition, title, and eligibility/title search matches.
Short-name regexes	curated abbreviations such as AML, NSCLC, DLBCL where safe	Avoid broad %aml%-style false positives.
MeSH / ancestor condition terms	CTGov search areas ConditionMeshTerm and ConditionAncestorTerm, plus maintained therapeutic-area mappings	Recover oncology/hematology trials whose raw condition text is weak.
Intervention type filter	CTGov InterventionType search/filter field and study JSON intervention module	Keep interventional therapeutic trials; default include DRUG, BIOLOGICAL, GENETIC, COMBINATION_PRODUCT, and source-reviewed RADIATION.
Study type filter	CTGov StudyType search/filter field and study JSON design/status modules	Default INTERVENTIONAL; observational studies only enter through explicit source/reference decisions.

Read-only legacy warehouse aggregates sampled 2026-07-23 show why this gate must use the official source enum instead of legacy strings: clinical_trials.study_type contains 93,594 INTERVENTIONAL, 8,555 Interventional, and 25,137 null rows; legacy trial_arm_interventions.intervention_type is dominated by drug and biological but also uses local labels such as combination and supportive. This is liveness/data-shape evidence, not source authority. The greenfield source adapter evaluates CTGov’s uppercase StudyType and InterventionType values; later normalization handles compatibility vocabulary.

Filtering outputs:

Asset / concept	Grain	Meaning
sources.ctgov.trial_discovery_runs	one CTGov search capture + query/filter/page boundary	The replayable source query contract: terms, filters, API version, data timestamp, scope-rule version, complete/bounded boundary, and actual page traversal.
sources.ctgov.trial_discovery_candidates	one candidate or rejected NCT id + filter run + matched rule	Candidate selected by disease/search-area/intervention/study-type filtering, or rejected with reason.
query.q_trial_discovery_candidates	one candidate + reason	Agent-facing diagnostics for why a trial entered or failed filtering.

Filtering rules:

• Filtering is a candidate-selection gate, not canonical disease truth. A trial selected by keyword still needs disease resolution and verifier/contract acceptance before downstream disease facts publish.
• Filter reasons must have provenance: CTGov API URL, query params, page token/rank when present, matched term/pattern/search area, intervention type, study type, source data timestamp, API version, and run id.
• False-positive and false-negative corrections change the versioned source-scope rule and replay discovery. Prior candidate decisions remain immutable evidence of what the old rule selected.
• External NCT references from labels, publications, regulators, or reviewed curation do not silently mutate this source capture. A future exact-id discovery input must emit the same source-candidate grain and provenance contract before it can fetch a snapshot.
• Exact CTGov snapshots, not discovery hits, are the source authority for published trial facts. Discovery candidates explain why we fetched or rejected a trial; sources.ctgov.trial_snapshots carries the immutable payload used downstream.

Asset Graph for This Slice

flowchart TD
  CTAPI[[ClinicalTrials.gov API]]
  ChiCTR[[ChiCTR scrape/import]]

  RawCTDiscovery[sources.ctgov.trial_discovery_runs / candidates]
  RawCT[sources.ctgov.trial_snapshots]
  CTDiffs[ops.ctgov_snapshot_diffs]
  RawChiCTR[sources.chictr.trial_snapshots]

  NormTrials[normalized.trial_records]
  NormResults[normalized.trial_result_records]
  NormSafety[normalized.trial_safety_event_records]
  EvidenceDesign[evidence.trial_design_claims]
  EvidenceResults[evidence.trial_result_claims]
  ClaimTables[evidence.trial_claim_tables]
  DeterministicEligibility[evidence.trial_claims]
  ReuseGate[provenance.legacy_llm_reuse_decisions]
  GapFill[evidence.trial_eligibility_gap_fill]
  EvidenceEligibility[(evidence.trial_eligibility_claims)]
  LegacyEligibility[[source-matched legacy eligibility]]
  Resolution[resolution.trial_entities]
  Curation[[curation.effective_overrides]]
  CanonicalTrials[canonical.trials]
  Query["q_trials family"]

  CTAPI --> RawCTDiscovery --> RawCT
  RawCT --> CTDiffs
  RawCT --> NormTrials
  RawCT --> NormResults
  RawCT --> NormSafety
  ChiCTR --> RawChiCTR --> NormTrials
  NormTrials --> EvidenceDesign --> Resolution
  NormResults --> EvidenceResults --> Resolution
  NormSafety --> EvidenceResults
  ClaimTables --> DeterministicEligibility
  ClaimTables --> GapFill
  NormTrials --> DeterministicEligibility --> EvidenceEligibility
  DeterministicEligibility --> ReuseGate
  LegacyEligibility --> ReuseGate
  ReuseGate -->|needs_fresh_llm only| GapFill --> EvidenceEligibility
  EvidenceEligibility --> Resolution
  Resolution --> CanonicalTrials --> Query
  Curation -. reviewed overrides .-> CanonicalTrials

The discovery run and candidate tables are distinct assets because they have distinct grains and downstream uses, but one HTTP traversal computes both. They therefore share one Dagster multi_asset computation and one database transaction. This is the intended Dagster abstraction for multiple assets computed by the same operation and upstream inputs: https://docs.dagster.io/api/dagster/assets#dagster.multi_asset.

Concrete assets

Layer	Asset key	Grain	Notes
source	sources.ctgov.trial_discovery_runs	one CTGov search capture + query/filter/page boundary	Persist source mode, complete/bounded scope, query params, API version, data timestamp, scope-rule version, page count, candidate count, and timing.
source	sources.ctgov.trial_discovery_candidates	one candidate/rejected NCT id + filter run + matched rule	Replaces hidden AACT SQL candidate filtering with replayable CTGov search provenance. Dagster materializes this table and its owning run together with one multi_asset computation.
source	sources.ctgov.trial_snapshots	one raw CTGov v2 study JSON per nct_id + pull/source data timestamp + content hash	Preserve complete source payload from /studies/{nctId} including protocol and results modules, source URL, API version, and source data timestamp.
ops	ops.ctgov_snapshot_diffs	one trial + module + prior/new snapshot pair	Compare module content hashes for protocol, eligibility, conditions, interventions, outcome measures/results, adverse events, baseline, participant flow, sponsors, locations, references. Drives targeted rematerialization.
source	sources.chictr.trial_snapshots	one content-addressed raw ChiCTR record/XML/HTML-derived snapshot	Preserve exact bytes, scrape metadata, and parser version. Repeated source version labels cannot collide; byte-identical transition rows are de-duplicated during normalization.
normalized	normalized.trial_records	one source-faithful trial per source trial id	Do not resolve drugs/diseases here; normalize source fields only.
normalized	normalized.trial_result_records	one source result object per CTGov outcome measure / group / statistic / analysis	CTGov result group ids such as OG000 remain source locators, not design-arm identity.
normalized	normalized.trial_safety_event_records	one CTGov adverse-event statistic per event group / serious-or-other section / AE term	Parse CTGov seriousEvents and otherEvents without collapsing them into endpoint rows.
normalized	normalized.trial_safety_event_total_records	one CTGov adverse-event section total per event group / total type	Parse deaths/serious/other affected+at-risk totals from CTGov eventGroups.
evidence	evidence.trial_claim_tables	one DDL/store owner	Owns the shared trial evidence relations used by deterministic, legacy-salvage, and inference writers. Writers depend on it instead of recreating shared tables.
evidence	evidence.trial_claims	one deterministic trial evidence rebuild	Writes conservative design, condition, eligibility, result, and safety claims from current normalized source records.
provenance	provenance.legacy_llm_reuse_decisions	one current trial + inference task decision	Hard provider-call gate: not_required, reused, needs_fresh_llm, or rejected.
evidence	evidence.trial_eligibility_gap_fill	one explicitly selected bounded cohort run	Disabled by default; only needs_fresh_llm candidates can call the provider, and accepted outputs receive decision/projection receipts.
evidence	evidence.trial_result_claims	one claim per endpoint/outcome/arm result	Preserve statistics, units, time frames, group labels, CTGov JSON paths, and source group ids.
evidence	evidence.trial_eligibility_claims	one claim per disease/biomarker/stage/treatment-line eligibility assertion	Extract from free-text eligibility; later resolved/verified, with curation overlay only if disputed.
resolution	resolution.trial_entities	one candidate set per extracted term/subject	Covers interventions, diseases, biomarkers, organisations, endpoints. It depends explicitly on the drug, disease/biomarker, endpoint/AE, and organisation owner-name projections; DDL existence alone is not a population dependency.
canonical	canonical.trials	one canonical trial per source registry id, plus cross-registry link if proven	NCT and ChiCTR ids are source-scoped external ids.
query	query.q_trials*	agent-facing grains below	Materialized by SQL assets after checks pass.

CTGov source table contracts

These are source-layer tables: no resolved identity, no LLM output, no curation state.

Table	Grain	Required columns
sources_ctgov.trial_discovery_runs	one CTGov search/discovery capture	discovery_run_key, source_release, api_version, source_data_timestamp, source_mode, capture_scope, is_complete_source_capture, max_live_records, scope_rule_version, query_params jsonb, page_size, page_count, source_url, started_at, completed_at, candidate_count, ingested_at
sources_ctgov.trial_discovery_candidates	one NCT candidate/rejection reason from one discovery run	discovery_candidate_key, discovery_run_key, nct_id, candidate_state, scope_basis, matched_area, matched_piece, matched_term, study_type, intervention_types text[], has_results, rank, page_token, source_locator jsonb, snapshot_id, rejection_reason, ingested_at
sources_ctgov.trial_snapshots	one full CTGov v2 study JSON capture	snapshot_id, nct_id, source_version, api_version, source_data_timestamp, pulled_at, source_url, payload jsonb, immutable_uri, content_hash, retrieval_metadata jsonb, source_release, ingested_at
ops.ctgov_snapshot_diffs	one module hash comparison for one snapshot	snapshot_id, nct_id, module_name, module_hash, previous_snapshot_id, previous_module_hash, changed, computed_at

started_at and completed_at describe a successful persisted capture. Failure status and error details stay in the Dagster run/event log; duplicating a second mutable run-state machine in this source table would create conflicting ownership. The exact discovery body hash and immutable URI live in provenance.source_artifacts.

Candidate states are exactly accepted and rejected. Fetch failures belong to the Dagster run event log, while changed scope decisions appear in a new immutable discovery run rather than a synthetic supersession state. Current deterministic scope_basis values are condition_keyword_match, condition_mesh_term, condition_ancestor_term, title_keyword_match, rejected_non_onco_heme, rejected_non_interventional, and rejected_non_therapeutic. Exact external references and reviewed curation enter through their owned evidence/curation paths rather than pretending to be CTGov search results.

module_name is a stable local label whose implementation maps to one exact CTGov JSON module:

identification       -> protocolSection.identificationModule
status               -> protocolSection.statusModule
sponsors             -> protocolSection.sponsorCollaboratorsModule
conditions           -> protocolSection.conditionsModule
design               -> protocolSection.designModule
arms_interventions   -> protocolSection.armsInterventionsModule
planned_outcomes     -> protocolSection.outcomesModule
eligibility          -> protocolSection.eligibilityModule
locations            -> protocolSection.contactsLocationsModule
references           -> protocolSection.referencesModule
condition_browse     -> derivedSection.conditionBrowseModule
participant_flow     -> resultsSection.participantFlowModule
baseline             -> resultsSection.baselineCharacteristicsModule
outcome_measures     -> resultsSection.outcomeMeasuresModule
adverse_events       -> resultsSection.adverseEventsModule

CTGov parser mapping contract

CTGov v2 path	Greenfield target	Notes
protocolSection.identificationModule.nctId	trial_key, primary_registry_id, nct_id	trial_key = trial:ctgov:<NCTID>; never collapse with ChiCTR without accepted cross-registry link.
protocolSection.identificationModule.briefTitle / officialTitle	normalized.trial_records.brief_title / official_title	Source text, not evidence extraction output.
protocolSection.statusModule	trial status/date fields	Preserve source date type/precision where available.
protocolSection.designModule	study type, phase/design fields	Drives cohort filter and query fields; no clinical interpretation here.
protocolSection.sponsorCollaboratorsModule	sponsor/collaborator source org terms	Resolve later through organisation spine.
protocolSection.conditionsModule.conditions plus derivedSection.conditionBrowseModule when present	condition terms and browse/ancestor terms	Candidate-selection evidence only until disease resolution accepts a link.
protocolSection.armsInterventionsModule.armGroups	design arm source records	Planned design arms; not result groups.
protocolSection.armsInterventionsModule.interventions	intervention component source records	Resolve later through drug/target/non-drug concept spines.
protocolSection.outcomesModule	planned endpoint/outcome claims	Planned endpoints; distinct from reported results in resultsSection.
protocolSection.eligibilityModule.eligibilityCriteria	eligibility text / eligibility claims	Feed LLM extraction + verifier/quarantine before canonical promotion.
resultsSection.outcomeMeasuresModule.outcomeMeasures[]	normalized.trial_result_records / q_trial_outcomes	One row per measurement statistic; analysis rows stay separate.
resultsSection.outcomeMeasuresModule.outcomeMeasures[].groups[]	source result groups	Group ids/labels are source locators; arm link requires accepted basis.
resultsSection.outcomeMeasuresModule.outcomeMeasures[].analyses[]	analysis result rows	Preserve p-value, CI, HR/OR/RR-style statistics as field-provenanced facts.
resultsSection.adverseEventsModule.eventGroups[]	safety-event total rows	Death/serious/other affected+at-risk totals are totals, not AE-term rows.
resultsSection.adverseEventsModule.seriousEvents[] / otherEvents[]	safety-event observation rows	One row per event term + group stat + event type.
resultsSection.baselineCharacteristicsModule	normalized/canonical baseline measurements -> q_trial_baseline_measurements	Preserve source group, category, statistic, dispersion, denominator, and provenance.
resultsSection.participantFlowModule	normalized/canonical participant flow -> q_trial_participant_flow	Preserve period, group, milestone/withdrawal reason, count, and provenance.
protocolSection.referencesModule	trial-reference source records	Publication slice owns detailed publication evidence.

Source locators must be stable enough to debug a fact without AACT. Prefer JSON-path-ish locators:

{"source": "ctgov", "nct_id": "NCT02481830", "path": "$.resultsSection.adverseEventsModule.seriousEvents[0].stats[1]"}

Slice-Local Domain Model

This slice introduces a clinical-trial model with separate protocol design, eligibility, and reported results subgraphs. Do not collapse them into one wide table: legacy had separate clinical_trials, study_plan_arms, study_plan_components, participation_criteria, result_groups, result_outcomes, outcome_measurements, outcome_analyses, trial_endpoints, trial_outcome_measures, and trial_arm_outcomes because those are different grains (../bioloupe-data-gov/docs/data-model.md, ../bioloupe-data-gov/app/models/).

Entities and grains

Entity	Grain	Canonical key recommendation	Legacy pressure	Notes
Trial	one registry study	trial:{source}:{registry_id}; e.g. trial:ctgov:NCT04380636, trial:chictr:ChiCTR...	clinical_trials.collection_source, NCT/ChiCTR ids	Trial identity is source-scoped. Cross-registry duplicates require exact cross-reference, verifier acceptance, or reviewed curation override.
Trial version / snapshot	one source payload at a version/time and content hash	trial_snapshot:{source}:{registry_id}:{source_version_or_pulled_at}:{content_hash}	legacy versions JSONB and reconciliation flow	Source dates are not assumed unique; byte-different same-date captures append instead of colliding.
Design arm	one protocol/planned arm	design_arm:{trial_key}:{source_arm_id_or_hash}	study_plan_arms	Planned arms differ from result groups; preserve source labels.
Intervention component	one named component within a design arm	intervention_component:{design_arm_key}:{component_hash}	study_plan_components	A component may resolve to a Drug, NCIt concept, procedure, radiation, device, placebo, or unknown.
Endpoint / outcome measure	one planned or reported measure	endpoint_claim:{trial_key}:{source_measure_id_or_hash}	outcomes, trial_endpoints, endpoint_groups	Keep source title/time frame; optionally link to canonical endpoint catalog.
Result group	one reported-results analysis group	result_group:{trial_key}:{source_group_code}	legacy result_groups; CTGov OG000 / EG000-style group ids	Not automatically identical to a design arm. Requires source-exact link, verifier acceptance, or reviewed curation override.
Outcome result	one measurement/statistic for an endpoint and group	trial_outcome:{trial_key}:{endpoint_key}:{group_key}:{stat_hash}	outcome_measurements, outcome_analyses, trial_arm_outcomes	Preserve param type, value, CI, p-value, unit, dispersion.
Eligibility assertion	one structured eligibility fact for a trial/disease context	eligibility:{trial_key}:{disease_or_context}:{modifier}:{value_hash}	participation_criteria, trial_eligibility_criteria	Includes disease, subtype, stage, prior lines, biomarker, ECOG, age.
Sponsor / collaborator role	one organisation role on a trial over a source record	trial_org_role:{trial_key}:{role}:{org_term_hash}	clinical_trials:sponsors:link_to_organisations	Link to canonical organisation only after resolution.
Trial reference	one source publication/reference linked to trial	trial_reference:{trial_key}:{pmid_or_doi_or_hash}	clinical_trial_references, publications workflow	Publications slice owns detailed evidence extraction.

Invariants

• A canonical trial must have at least one source-scoped registry id and a provenance receipt to an immutable raw source snapshot.
• A q_trials row must not claim a resolved drug/disease/biomarker/organisation unless the corresponding resolution subject has an accepted deterministic/LLM verifier basis, exact allowed basis, or reviewed curation override.
• Protocol design arms and reported result groups are separate grains. Matching them is a resolution/verifier subject, not a join assumption; curation can override disputed matches.
• Endpoint names from sources are evidence, not canonical endpoint truth. Canonical endpoint linkage is optional and accepted through resolution/verifier policy, with curation overrides for disputed mappings.
• Eligibility is contextual: a trial can have multiple disease contexts, subtypes, stages, biomarker constraints, treatment-line constraints, and inclusion/exclusion polarity.
• ChiCTR and ClinicalTrials.gov ids are external identifiers, not interchangeable canonical ids.

Slice-Local Entity Resolution

Legacy got this right in spirit but too imperatively: terms were gathered, searched, LLM-ranked, judged, then post-processed. The greenfield model should keep the cascade as resolution assets + verifier artifacts + optional curation override subjects, not as hidden workflow side effects (../bioloupe-data-gov/app/workflows/clinical_trial_eligibilities_workflow.rb).

Resolution subjects

Subject kind	Source term examples	Candidate target types	Publishable basis	High-impact?
trial_intervention_link	pembrolizumab, KEYTRUDA, MK-3475, chemo regimen, placebo	drug spine resolved_drug_key; NCIt non-drug concept; combination/regimen; unresolved	exact synonym, cross-reference, context-verified, verifier-passed fuzzy/LLM, reviewed curation override	yes
trial_disease_link	NSCLC, breast cancer, relapsed AML	canonical Disease / NCIt disease concept	NCIt exact/cross-reference, context-verified, verifier-passed semantic/LLM, reviewed curation override	yes
trial_biomarker_link	HER2-positive, EGFR exon 20 insertion, PD-L1 TPS ≥ 50%	canonical Biomarker / gene/protein/variant concept	exact/cross-reference/context-verified or reviewed curation override	yes
trial_endpoint_link	ORR, PFS, OS, DOR, adverse event rate	canonical Endpoint catalog	exact abbreviation mapping, accepted endpoint synonym, verifier-passed LLM	medium
trial_safety_ae_term_link	neutropenia, febrile neutropenia, hepatic toxicity	canonical adverse-event term / CTCAE-MedDRA-aligned concept	exact vocabulary mapping, source vocabulary, verifier-passed semantic/LLM, reviewed curation override	medium/high for safety analytics
trial_org_link	sponsor/collaborator free text	canonical Organisation	exact legal name/alias, reviewed fuzzy override, reviewed curation override	medium/high when used for ownership/sponsor analytics
trial_result_group_arm_link	CTGov result group label vs design arm title	design arm	deterministic id if source supplies; otherwise verifier or reviewed curation override	high for outcomes/SOC evidence
cross_registry_trial_link	NCT and ChiCTR records possibly same study	canonical trial identity cluster	exact registry cross-reference, verifier, or reviewed curation override	high

Resolution cascade

flowchart LR
  Term[Extracted source term]
  Exact[Exact / synonym lookup]
  XRef[External-id or source cross-reference]
  Search[Semantic / lexical candidates]
  Judge[Independent verifier / judge]
  Basis[Accepted resolution basis]
  Accepted[Accepted canonical link]
  Conflict[Conflict / curation candidate]

  Term --> Exact
  Exact -- hit --> Basis
  Exact -- no hit --> XRef
  XRef -- hit --> Basis
  XRef -- no hit --> Search --> Judge --> Basis
  Basis -- accepted --> Accepted
  Basis -- ambiguous/rejected --> Conflict

Resolution outputs are candidate sets with explicit provenance basis; accepted links are derived from exact/cross-reference/context-verifier policy, not from curation by default. No fuzzy top hit auto-publishes. Curation enters only when a curator agent/human reviews an override or suppression under 05-agentic-curation-and-overrides.md.

Evidence Claim Schemas

Evidence claims are source-grounded records before canonical promotion. They should be narrow and replayable.

trial_design_claim

Field	Meaning
claim_id	stable id generated from source snapshot + extractor version + claim path
trial_key	source-scoped trial key
source_locator	CTGov API URL + JSON path, or ChiCTR artifact/html locator
claim_kind	arm, intervention_component, dose, route, schedule, investigational_component, sponsor_role
raw_text	text span or source field used
claim_value	structured JSON value
extractor_version	deterministic parser/LLM/prompt/schema version
requires_resolution	boolean + subject kind

trial_eligibility_claim

Field	Meaning
trial_key	source-scoped trial key
polarity	inclusion / exclusion / unknown
disease_context_term	raw disease context if present
modifier	disease, subtype, extent, stage, status, risk, treatment_setting, prior_treatment_lines, treatment_lines, biomarker, age, ECOG
operator	eq, neq, gt, gte, lt, lte, range, textual
value_text	source-faithful textual value/span
value_num	scalar numeric threshold; null for ranges
value_num_min, value_num_max	explicit numeric bounds; both required for range, with the operator carrying inclusive/strict meaning
value_unit	years, line, ECOG, or another source-grounded unit
evidence_text	source criterion sentence/span
inference_decision_id, inference_projection_revision	accepted inference provenance; null for deterministic/legacy claims
resolution_subject_key	optional key for disease/biomarker resolution

Legacy pressure: TrialEligibilityCriterion already separated operator/modifier/value/range/context and had modifiers for subtype, extent, stage, status, risk, treatment setting, treatment lines, and disease (../bioloupe-data-gov/app/models/trial_eligibility_criterion.rb).

trial_result_claim

Field	Meaning
trial_key	source-scoped trial key
source_artifact_key	immutable CTGov/ChiCTR source snapshot key
source_locator	CTGov JSON path / ChiCTR artifact locator
source_result_group_key	CTGov result group id such as OG000 / EG000, or source-scoped ChiCTR group id if present
endpoint_source_key	source outcome/endpoint id or hash
measure_title	source measure title
time_frame	source time frame
statistic	param type/value/unit/dispersion/CI/p-value as available
population	analyzed population / subgroup if present
arm_link_subject_key	optional subject for result-group-to-design-arm resolution

Curation Override Subjects

The curation store defined in 05-agentic-curation-and-overrides.md is reused for reviewed corrections and overrides, not routine extraction/resolution. Clinical trials adds these high-value override subject kinds:

Subject kind	Why overrideable	Example curation decision
trial_intervention_link	Wrong drug links poison every downstream program view.	“MK-3475 in NCT… resolves to pembrolizumab/Drug X, basis exact synonym.”
trial_disease_link	Trial conditions and eligibility disease context drive disease pipelines.	“advanced NSCLC resolves to Non-Small Cell Lung Cancer, not generic Lung Cancer.”
trial_biomarker_link	Biomarker-positive eligibility changes treatment context.	“HER2 overexpressing maps to HER2-positive biomarker context.”
trial_endpoint_link	Endpoint normalization affects efficacy comparisons.	“objective response maps to ORR endpoint.”
trial_safety_ae_term_link	Safety-event term normalization affects safety comparisons and adverse-event analytics.	“febrile neutropenia resolves to the accepted AE term, not a generic neutropenia bucket.”
trial_result_group_arm_link	Result groups must align to the correct arm before outcome values are trusted.	“CTGov result group B corresponds to study plan arm pembrolizumab + chemotherapy.”
cross_registry_trial_link	Cross-registry duplicate merging changes canonical trial identity.	“ChiCTR… and NCT… are / are not the same study.”
eligibility_modifier_assertion	Treatment line/stage/subtype claims are often ambiguous.	“second-line or later becomes treatment_lines min=2.”

Curator agents may propose these changes, but high-impact overrides require human review. Verifier assets should handle the default long tail; curation exists for escalations, sampled audits, disputes, suppressions, and high-impact reversals.

Canonical Assets

Canonical clinical-trials assets should be pragmatic relational tables/materialized assets, not a formal ontology clone.

Canonical asset	Grain	Built from
canonical.trials	one canonical trial identity	accepted source trial snapshots + cross-registry links
canonical.trial_status_observations	one status/phase observation per retained source snapshot	immutable registry snapshots; capture time is not asserted effective time
canonical.trial_sites	one registry-listed site	source location/contact records; contacts are not organisation identity
canonical.trial_design_arms	one design arm	accepted design-arm evidence claims
canonical.trial_intervention_components	one intervention component in a design arm	accepted intervention claims + resolution/verifier receipts
canonical.trial_conditions	one disease/condition assertion per trial/context	accepted disease resolution/verifier receipts
canonical.trial_eligibility_assertions	one eligibility modifier assertion	eligibility claims + resolution/verifier receipts; reviewed curation overlays when present
canonical.trial_endpoints	one endpoint/measure assertion	outcome/endpoint claims + endpoint resolution/verifier receipts
canonical.trial_result_groups	one source result group, optionally linked to design arm	result group claims + arm-link resolution/verifier receipts
canonical.trial_outcomes	one statistic/measurement for endpoint/group/subgroup	result claims + contract checks
canonical.trial_safety_events	one adverse-event term observation for one trial result group and event type	CTGov adverse-event rows + AE-term resolution + arm-link receipts
canonical.trial_safety_event_totals	one adverse-event total/denominator for one trial result group and event type	CTGov adverse-event group totals + result-group link receipts
canonical.trial_participant_flow	one count per source result group/period/milestone or withdrawal reason	CTGov participant-flow module
canonical.trial_baseline_measurements	one baseline statistic per source result group/measure/category	CTGov baseline-characteristics module
canonical.trial_organisation_roles	one sponsor/collaborator role	organisation resolution/verifier receipts
canonical.trial_references	one publication/reference link	source references + publication-link evidence

q_trials Family

All tables below are read-only SQL models in the agent query layer. They must appear in generated q_catalog, and provenance-required rows/facts must resolve through q_provenance.

The seventeen models are one consistency family, not seventeen independently refreshed views. Dagster exposes one non-subsettable multi-asset. It freezes every checked-in SELECT, validates the complete candidate against schema/provenance/key constraints and owner spines, then atomically repoints every stable view plus the clinical_trials serving head. A failed candidate leaves the prior complete generation served; rollback also operates on the whole family. See 07 and 08 for the shared protocol.

Query model	Grain	Purpose
q_trials	one canonical trial	headline trial search: registry ids, title, phase, status, start/completion dates, source coverage, sponsors, high-level disease/drug summary
q_trial_discovery_candidates	one discovered or rejected trial candidate	diagnostics for why a trial entered/failed cohort filtering; owned by clinical trials, not the global search slice
q_trial_interventions	one intervention component per design arm	agent questions about drugs/regimens tested, monotherapy vs combination, investigational flag, dose/schedule summary
q_trial_design_arms	one design arm	arm-level protocol design, arm type, title, description, component count, review status
q_trial_conditions	one disease/condition context per trial	disease-specific trial discovery and disease pipeline rollups
q_trial_eligibility	one eligibility assertion	structured inclusion/exclusion constraints for disease, stage, biomarker, prior therapy, line, ECOG/age
q_trial_endpoints	one endpoint/measure assertion	source endpoint title/time frame + canonical endpoint link when accepted
q_trial_outcomes	one endpoint/group/statistic row	arm-level or group-level reported efficacy/result-measure values with provenance
q_trial_safety_events	one adverse-event term observation per trial/result group/event type	arm-level or group-level CTGov safety counts with AE-term resolution
q_trial_safety_event_totals	one adverse-event total per trial/result group/event type	denominators and total affected counts for serious/other adverse event sections
q_trial_result_groups	one CTGov result group + accepted design-arm linkage state	diagnostic/support surface for arm-link explainability; normal clinical questions use q_trial_outcomes / q_trial_safety_events / q_trial_design_arms (ADR-0005)
q_trial_status_history	one immutable registry status/phase observation	time-aware history behind the current q_trials projection
q_trial_sponsors	one sponsor/collaborator role	preserve source org term and publish resolved_org_key only on accepted basis
q_trial_sites	one registry-listed site	recruitment status, geography, source contacts, and provenance
q_trial_references	one registry-stated reference	source PMID/citation bridge; accepted publication identity/link remains owned by docs/23
q_trial_participant_flow	one participant count per source group/period/milestone or withdrawal reason	participant disposition without flattening periods or result groups
q_trial_baseline_measurements	one baseline statistic per source group/measure/category	baseline population context without collapsing result-group grain
q_provenance receipts, optionally exposed through q_trial_provenance	one or more receipts per provenance-required row/fact	global provenance back to source snapshots, extraction runs, evidence claims, resolution/verifier receipts, and curation overlays when applicable; q_trial_provenance may exist only as a convenience view over q_provenance

Seed q_trials columns

trial_key
primary_registry_source
primary_registry_id
nct_id
chictr_id
brief_title
official_title
phase
trial_status
study_type
start_date
primary_completion_date
completion_date
disease_names[]
resolved_disease_keys[]
intervention_names[]
resolved_drug_keys[]
sponsor_names[]
resolved_sponsor_keys[]
has_results
has_structured_eligibility
has_unresolved_high_impact_conflicts
source_freshness_at
provenance_key

Downstream key contract

resolved_trial_key(s) used by downstream slices (trial-event mentions in 24-slice-news-intelligence.md, trial-derived evidence in 23-slice-publications.md and 25-slice-standard-of-care.md) is defined here as a stable key into q_trials — the same identity as trial_key (e.g. trial:ctgov:NCT04380636). Downstream slices reference this key; they do not mint trial identity, and cross-registry duplicates still require exact cross-reference, verifier acceptance, or reviewed curation override before two registry keys collapse. resolved_sponsor_keys is not a separate identity: it is the set of resolved_org_key(s) from 26-slice-organisation-intelligence.md observed in a sponsor/collaborator role on this trial, so q_trials.resolved_sponsor_keys must resolve into the organisation spine rather than create a parallel sponsor namespace.

q_trial_outcomes comparative fact contract

q_trial_outcomes is the clinical-trials side of the shared therapeutic-fact model in 20-pipeline-families.md. It is a registry/source-reported result fact, not a publication result and not a label indication. ClinicalTrials.gov v2 separates protocol modules from resultsSection.outcomeMeasuresModule, participantFlowModule, baselineCharacteristicsModule, and adverseEventsModule; the greenfield row grain follows that source pressure rather than collapsing everything into a wide trial row (ClinicalTrials.gov study data structure: https://clinicaltrials.gov/data-api/about-api/study-data-structure; OpenAPI: https://clinicaltrials.gov/api/oas/v2).

Contract field	Required meaning
Purpose	Answer: “What result was reported for this trial endpoint/group/subgroup, and why can I trust it?”
Grain	One reported statistic/measurement for one trial_key + source endpoint/outcome + source result group + population/subgroup + statistic kind.
Stable row key	trial_outcome:{trial_key}:{endpoint_source_key}:{source_result_group_key}:{population_hash}:{statistic_kind}.
Required columns	trial_outcome_key, trial_key, primary_registry_id, endpoint_source_key, endpoint_term, canonical_endpoint_key, source_result_group_key, design_arm_key, arm_link_state, population_label, population_n, statistic_kind, statistic_value, statistic_unit, dispersion_type, dispersion_value, ci_lower, ci_upper, p_value, time_frame, source_family, source_artifact_key, source_locator, provenance_key.
Search affordances	filters on trial_key, registry id, canonical endpoint, intervention/drug keys via linked design arm, disease keys via trial context, has-results flag, statistic kind; trigram on endpoint/group labels.
Provenance rule	provenance_grain=bundle for every row; field receipts for statistic_value, p_value, ci_lower, ci_upper, population_n, and arm_link_state when present.
Refresh/materialization	immutable table in the atomically promoted clinical_trials generation, served through stable query.q_trial_outcomes; the complete seventeen-model family moves only after canonical, endpoint, arm-link, provenance, and contract gates pass.
Blocking checks	schema_contract, provenance_coverage, source_artifact_immutability, source_locator, resolution_basis, llm_verifier_required, curation_review_state when override applies.

Allowed fact_path examples:

trial_outcome:trial=NCT01234567:endpoint=orr:group=experimental:attr=statistic_value
trial_outcome:trial=NCT01234567:endpoint=pfs:group=control:attr=ci_lower
trial_outcome:trial=NCT01234567:endpoint=orr:group=experimental:attr=arm_link_state
trial_safety_event:trial=NCT01234567:event=neutropenia:event_type=serious:group=experimental:attr=subjects_affected
trial_safety_event_total:trial=NCT01234567:event_type=other:group=control:attr=subjects_at_risk

Publication rows may support or challenge this row, but they do not overwrite it. Cross-source comparisons happen through canonical assertions and provenance bundles, not by treating publication and registry statistics as the same row.

q_trial_outcomes column contract seed

Column	Type	Nullability	Semantic type	Provenance
trial_outcome_key	text	not null	query row key	row
trial_key	text	not null	canonical trial key	row
primary_registry_id	text	not null	source-scoped external id	row
endpoint_source_key	text	not null	source endpoint id/hash	bundle
endpoint_term	text	not null	source endpoint text	bundle
canonical_endpoint_key	text	nullable	resolved endpoint key	field when present
source_result_group_key	text	not null	source result group key	bundle
source_result_group_label	text	nullable	source result group label	bundle
design_arm_key	text	nullable	linked protocol arm key	field when present
arm_link_state	text	not null	resolution state	field
population_label	text	nullable	analyzed population/subgroup	bundle when present
population_n	numeric	nullable	count	field when present
statistic_kind	text	not null	outcome statistic kind	bundle
statistic_value	numeric	nullable	numeric result	field when present
statistic_text	text	nullable	non-numeric result text	field when present
statistic_unit	text	nullable	unit	field when present
dispersion_type	text	nullable	sd/se/range/etc	field when present
dispersion_value	numeric	nullable	dispersion value	field when present
ci_lower	numeric	nullable	confidence interval lower	field when present
ci_upper	numeric	nullable	confidence interval upper	field when present
p_value	numeric	nullable	p-value	field when present
time_frame	text	nullable	source time frame	bundle when present
source_family	text	not null	source family	row
source_artifact_key	text	not null	immutable source artifact	bundle
source_locator	jsonb	not null	source locator	bundle
source_authority_tier	text	not null	authority tier	row
assertion_state	text	not null	effective assertion state; curation overlay if applicable	row
provenance_key	uuid	not null	provenance bundle	bundle

Catalog notes: arm_link_state allowed values should include source_exact, curation_override, verifier_accepted, unlinked, and conflict. Numeric fields are nullable because some registry outcomes are categorical/textual, but at least one of statistic_value or statistic_text must be present for a published row.

q_trial_outcomes vocab/search/example seed

Legacy data-shape pressure from the local datalake confirms the query contract needs normalization rather than raw enum passthrough: result_outcomes.outcome_type contains PRIMARY, SECONDARY, OTHER_PRE_SPECIFIED, and POST_HOC; result/measurement param_type values include NUMBER, COUNT_OF_PARTICIPANTS, MEAN, MEDIAN, GEOMETRIC_MEAN, LEAST_SQUARES_MEAN, COUNT_OF_UNITS, and GEOMETRIC_LEAST_SQUARES_MEAN; dispersion values include standard deviation, standard error, confidence intervals, full range, and inter-quartile range. CTGov v2 is the source payload; these legacy AACT-derived values are compatibility pressure for the stable q_* vocabulary, not proof that AACT remains a source spine. The legacy tables also show why p_value, confidence intervals, and hazard/odds/risk-ratio style analysis fields must be field-provenanced rather than hidden in JSON (public.result_outcomes, public.outcome_measurements, public.outcome_analyses, sampled from local datalake).

Controlled vocabularies:

Field	Controlled values / normalization
outcome_type if exposed later	primary, secondary, other_pre_specified, post_hoc mapped from CTGov/legacy uppercase source values.
statistic_kind	count_of_participants, count_of_units, number, mean, median, geometric_mean, least_squares_mean, geometric_least_squares_mean, hazard_ratio, odds_ratio, risk_ratio, p_value, textual.
dispersion_type	standard_deviation, standard_error, confidence_interval_95, confidence_interval_90, confidence_interval_other, full_range, inter_quartile_range, geometric_coefficient_of_variation, other.
arm_link_state	source_exact, curation_override, verifier_accepted, unlinked, conflict.
source_authority_tier	registry_result. Protocol endpoint definitions live in q_trial_endpoints; derived statistics require a future versioned model rather than an unused enum slot.

Search/index contract:

Search need	Required index/affordance
Fetch all outcomes for one trial	btree on trial_key; btree on primary_registry_id.
Filter by endpoint/statistic	btree on canonical_endpoint_key, statistic_kind; trigram or full-text on endpoint_term.
Find unresolved arm links	partial btree on arm_link_state where arm_link_state in ('unlinked','conflict').
Source/provenance audit	btree on source_family, source_artifact_key; hash/btree on provenance_key.
Numeric comparison	btree on (canonical_endpoint_key, statistic_kind, statistic_value) where statistic_value is not null.

q_trial_safety_events contract seed

Field	Required content
Purpose	Registry-reported adverse-event observations from ClinicalTrials.gov results.
Grain	One trial + source result group + event type + organ system + adverse-event term.
Stable row key	trial_safety_event_key (trial_safety_event:<trial_key>:<source_result_group_key>:<event_type>:<semantic_hash>), used by q_provenance.q_row_key; source locators stay in source_locator.
Required columns	trial_safety_event_key, trial_key, primary_registry_id, source_result_group_key, source_result_group_label, design_arm_key, arm_link_state, event_type, organ_system, adverse_event_term, resolved_ae_term_key, subjects_affected, subjects_at_risk, event_count, percentage, time_frame, assessment, vocab, source_family, source_artifact_key, source_locator, source_authority_tier, assertion_state, provenance_key.
Search affordances	btree on trial_key, primary_registry_id, resolved_ae_term_key, event_type, source_result_group_key; trigram/full-text on adverse_event_term and organ_system; partial index on unresolved/conflict arm links.
Provenance rule	provenance_grain=bundle for row identity and source locator; field receipts for subjects_affected, subjects_at_risk, event_count, percentage, and arm_link_state.
Blocking checks	schema, provenance coverage, source artifact immutability, source locator, non-negative counts, subjects_affected <= subjects_at_risk when both present, percentage recomputation tolerance, result-group referential integrity, AE-term referential integrity when resolved.

event_type maps ClinicalTrials.gov serious/other sections to serious and other. Do not collapse that source seriousness section with grade_gte3: seriousness is an outcome/regulatory classification, while CTCAE grade is severity. The term and grade vocabularies come from 32; the source event section remains trial-owned.

q_trial_safety_event_totals contract seed

Field	Required content
Purpose	Registry-reported denominator/total rows for adverse-event sections.
Grain	One trial + source result group code + event type + classification.
Stable row key	trial_safety_event_total_key (trial_safety_event_total:<trial_key>:<source_result_group_key>:<event_type>:<semantic_hash>); source locators stay in source_locator.
Required columns	trial_safety_event_total_key, trial_key, primary_registry_id, source_result_group_key, source_result_group_label, design_arm_key, arm_link_state, event_type, classification, subjects_affected, subjects_at_risk, percentage, source_family, source_artifact_key, source_locator, source_authority_tier, assertion_state, provenance_key.
Provenance rule	Field receipts for affected/risk/percentage and arm-link state.
Blocking checks	same count/range/source-locator gates as q_trial_safety_events; totals are not interchangeable with individual AE-term rows.

Example row shape:

trial_outcome_key: trial_outcome:trial:ctgov:NCT01234567:orr:grp-exp:overall:count_of_participants
trial_key: trial:ctgov:NCT01234567
primary_registry_id: NCT01234567
endpoint_term: Objective Response Rate
canonical_endpoint_key: endpoint:orr
source_result_group_key: ctgov_group:EG000
source_result_group_label: Experimental Arm
arm_link_state: verifier_accepted
population_label: Overall analyzed population
population_n: 84
statistic_kind: count_of_participants
statistic_value: 28
statistic_unit: Participants
time_frame: up to 24 months
source_family: ctgov
source_authority_tier: registry_result
provenance_key: <uuid>

Pathological examples the implementation must handle:

Case	Why it is dangerous	Required behavior
Result group cannot be mapped to a protocol arm	CTGov result groups often do not have a safe one-to-one design arm link; forcing a match corrupts drug/arm analytics.	Publish the outcome only with design_arm_key is null and arm_link_state='unlinked'; block any row that claims a design arm without accepted basis.
Same endpoint has numeric result and separate analysis row	Measurements and analyses can carry different but related facts: count/percentage in measurements, HR/p-value/CI in analyses.	Preserve separate field provenance for numeric measurement, HR/OR/RR, p-value, and CI; do not hide analyses inside one untraceable JSON blob.
Publication reports a different ORR from registry	Publication evidence may update, refine, or conflict with registry result values.	Keep registry result in q_trial_outcomes; publication value remains in q_publication_outcomes; optional comparison marks conflict without overwriting either.
Outcome is textual/categorical only	Some source outcomes are not numeric but are still meaningful.	Allow statistic_text with statistic_value is null, but require source locator and statistic_kind='textual'.

Checks and Quality Gates

Blocking checks:

• every canonical.trials row has at least one source snapshot provenance pointer;
• every provenance-required q_trials row/fact has at least one q_provenance receipt;
• every resolved drug/disease/biomarker/org key in q_* has an accepted resolution/verifier receipt, allowed deterministic basis, or reviewed curation override;
• every resolved_drug_key(s), resolved_disease_key(s), resolved_biomarker_key(s), canonical_endpoint_key, and resolved_sponsor_keys (org) value resolves to an active row in its owner q_* model (referential_integrity gate in 08-contracts-and-quality.md); resolved_sponsor_keys resolves through the organisation spine, not a local sponsor namespace;
• no high-impact unresolved conflict publishes as accepted truth;
• q_trial_outcomes rows preserve source group/endpoint/statistic identifiers and cannot be orphaned from trial + source snapshot;
• result-group-to-design-arm links publish only if exact/source-provided, verifier-passed, or reviewed curation override;
• schema shape matches q_catalog.

Warning checks:

• source freshness drift;
• unusual count deltas by source/partition;
• high unresolved-candidate rate for a source run;
• ChiCTR scraper degradation / parser drift;
• LLM extraction output distribution shift.

Operations

• Cadence: ClinicalTrials.gov API spine should refresh on a stable cron aligned to source update rhythm; legacy used Tuesday 08:00 UTC (../bioloupe-data-gov/docs/clinical-trials.md). Capture CTGov /version dataTimestamp with every run.
• Partitioning: source snapshots should partition by source-native window/release where replayable. Canonical rebuilding may target trial keys, but the stable query family currently publishes as one complete non-subsettable generation.
• Backfills: backfill by source/date partition and by trial-key subsets, not full global recompute.
• Curation sensor: new reviewed override decisions for intervention/disease/biomarker/arm-link subjects trigger the owning canonical rebuild and then one complete atomic q_trials family publication.
• Retries: narrow retries for transient CTGov/ChiCTR/network/LLM errors; no swallow-and-continue.
• Eligibility inference: normal graph/bootstrap runs do not select evidence/trial_eligibility_gap_fill and make no provider calls. An operator selects it explicitly, enables it in run config, and uses max_requests plus optional trial_keys as the cohort boundary. The default is five requests and a retry-inclusive $10 preflight ceiling. Active task contract trial_eligibility_extract@v6 uses native structured output, exact declared-artifact quotes, requested-facet scope, deterministic section-bound polarity, quote-bound text/numbers, and projection trial-eligibility-projection-v6.
• AACT reconciliation: optional, non-publishing check only. It may compare CTGov-derived counts/coverage to AACT samples, but failures create investigation alerts; they do not make AACT source authority.
• ChiCTR: if retained in v1, scraper health metrics are first-class: zero-result anomaly, Cloudflare/anti-bot detection, parser drift, keyword coverage, S3/raw-artifact completeness (../bioloupe-data-gov/docs/chictr.md).

ops:
  trial_eligibility_gap_fill:
    config:
      enabled: true
      max_requests: 5
      max_cost_usd: 10.0
      trial_keys:
        - trial:ctgov:NCT00000000

The asset still refuses any row whose latest reuse decision is not needs_fresh_llm. A current source-artifact change or projection-revision change creates new work identity; an accepted replay uses the inference cache and immutable projection receipt.

Provider activation additionally requires all checked-in eligibility semantic cases to pass the pinned active profile. Offline goldens currently prove validators and projection semantics only; no paid provider-backed eligibility evaluation has been run.

Testing Strategy for This Slice

Minimum tests:

1. sources.ctgov.trial_snapshots fixture uses real CTGov v2 shape and records API version, source data timestamp, source URL, content hash, and immutable artifact key.
2. ChiCTR snapshot identity includes the full content hash, and normalization collapses only byte-identical observations with the same source version and capture time.
3. CTGov discovery tests prove the source asset records accepted and rejected NCT candidates with query/filter/search-area provenance.
4. normalized.trial_records preserves source ids and does not invent resolved entities.
5. Trial identity key generation handles NCT and ChiCTR ids without collision.
6. CTGov resultsSection.outcomeMeasuresModule parses outcome measurements and analyses into separate rows with field provenance for value, CI, p-value, and statistic kind.
7. CTGov resultsSection.adverseEventsModule parses serious/other events plus event-group totals into q_trial_safety_events and q_trial_safety_event_totals.
8. Intervention resolution exact synonym wins; fuzzy/semantic candidate without verifier does not publish.
9. Disease subtype more-specific-than-base case is represented without losing base context (legacy had special subtype logic in ParticipationCriterion#all_diseases).
10. Eligibility fixtures cover polarity, facets, scalar thresholds, explicit two-bound ranges, and rejected reversed/incomplete ranges.
11. A FakeProvider plus local Postgres proves the enabled eligibility path, lifecycle receipts, accepted projection, cache replay, and no second provider call.
12. Result group and design arm are not joined unless accepted by deterministic source evidence, verifier acceptance, or reviewed curation override.
13. Curation supersession changes exactly the affected canonical/q_* rows.
14. q_trials, q_trial_interventions, q_trial_eligibility, q_trial_outcomes, q_trial_safety_events, and q_trial_safety_event_totals satisfy generated q_catalog and required q_provenance contracts.
15. Thin end-to-end fixture: one CTGov v2 NCT trial with two arms, one drug synonym, one disease, one biomarker eligibility, one endpoint, one result group, one safety event, one curation override.
16. The coherent fixture resolves NCT04380636 to drug:pembrolizumab and disease:non-small-cell-lung-cancer-nsclc; a graph test proves trial resolution depends on the populated owner-name assets and never queries obsolete plural owner tables.

Concepts Promoted to 20-pipeline-families.md

Stable cross-slice concepts discovered here:

• Source-scoped external id — registry ids are namespaced; do not treat an NCT id and ChiCTR id as the same key type.
• Protocol-vs-result grain separation — planned design arms and reported result groups are different things until linked.
• Evidence claim before canonical assertion — source text/table rows become claims, then resolution/verifier state, then canonical facts; curation only overlays reviewed corrections.
• Controlled vocabulary as anchor, not prison — NCIt/CDISC/MeSH guide modelling but do not force ontology-native storage.

Deliberate Boundaries

• ChiCTR: model as a first-class source identity now, but implement scraper hardening after the CTGov NCT spine unless product priority demands immediate operational parity. No NCT assumption may leak into canonical/query tables.
• ClinicalTrials.gov history: capture forward history through immutable source snapshots, content hashes, and ops.ctgov_snapshot_diffs. Pre-existing historical archive ingestion is a separate source family decision, not an AACT fallback.
• CDISC/SDTM: use as a naming and interchange reference only. Do not clone SDTM domains or force the q-layer into an SDTM-shaped schema.
• AACT: use only for reconciliation, parity audits, or legacy coverage investigation. It must not be an upstream dependency for canonical.* or q_* facts.

Implementation Contract

The CTGov NCT spine implementation contract is now:

1. Source mode is explicit: deterministic fixture mode for tests; opt-in live mode through ctgov_source Dagster resource run config. max_live_records may bound live smoke runs and must be reflected in materialization metadata.
2. One Dagster multi_asset capture writes sources.ctgov.trial_discovery_runs and sources.ctgov.trial_discovery_candidates atomically; it never performs the search twice to materialize the two grains.
3. Each discovery run records query params, CTGov API version, source data timestamp, source mode, complete/bounded capture boundary, max_live_records, deterministic scope-rule version, actual page count, and candidate count. Its key covers the query, boundary, upstream version, rule version, and capture attempt.
4. Candidate selection deterministically requires an oncology/hematology source term, INTERVENTIONAL study type, and at least one supported therapeutic intervention type. Accepted and rejected rows retain the exact matched source term/path and stable reason code.
5. Bounded smoke captures remain durable diagnostics but do not replace the latest complete cohort in q_trial_discovery_candidates and do not rewrite its assertion-provenance links.
6. sources.ctgov.trial_snapshots stores immutable content-addressed v2 study JSON per accepted NCT id and registers source artifacts with hashes, URL, API version, source data timestamp, and retrieval metadata. A byte-different payload at the same CTGov source date creates a new row.
7. ops.ctgov_snapshot_diffs records per-module hashes for protocol, eligibility, conditions, arms/interventions, outcome measures, adverse events, baseline, participant flow, sponsors, locations, and references.
8. normalized.trial_records parses CTGov protocol modules into source-scoped trials, arms, interventions, sponsors, conditions, browse terms, and eligibility text.
9. CTGov result parsers flatten outcomeMeasuresModule, participantFlowModule, baselineCharacteristicsModule, and adverseEventsModule into normalized result, participant-flow, baseline, safety-event, and safety-total records.
10. Canonical/query publication preserves source-native grains and receipts across all seventeen models, including q_trial_sponsors, q_trial_sites, q_trial_references, q_trial_participant_flow, and q_trial_baseline_measurements; one validated generation promotes or rolls back as a unit, and publication identity remains owned by docs/23.
11. evidence/trial_claim_tables exclusively owns shared trial-evidence DDL; deterministic, legacy-salvage, and inference writers depend on that asset.
12. Eligibility provider work is opt-in and bounded. Only current-source needs_fresh_llm rows can execute, and only exact-quote, requested-facet, numerically coherent accepted outputs can be projected.
13. q_trial_eligibility preserves disease context and resolved disease/biomarker keys, uses value_num only for scalar thresholds, and exposes both numeric bounds for ranges.