10 Testing Strategy

Testing Strategy

Status: IMPLEMENTED FIXTURE/PACKAGING GATES / LIVE UNPROVEN. Unit, isolated-Postgres, loaded-definition, installed-wheel, blank-database, P01-P12 fixture, locked UI/gateway, production image, and disposable full-Compose gates are implemented. Paid semantic model evaluation, source-specific real-network suites, production predecessor upgrade/restore, and consumer compatibility remain separate and incomplete. Clinical-trials last-known-good failure injection is implemented; other query families are not generation-protected. Denoising state: converged (noise -> rough -> sharp -> converged) Updated: 2026-07-25

Purpose

Define how the built repo proves correctness: from pure transformation logic through Dagster definitions, asset checks, SQL contracts, LLM extraction/enrichment invariants, curation override/review invariants, and end-to-end slice smoke tests.

Test pyramid (decided)

1. Unit — pure functions in lib/ (normalizers, parsers, precedence/merge logic) tested in isolation.
2. Asset tests — invoke pure assets directly where possible and materialize graph subsets in-process with fake resources for dependency-heavy assets. Dagster’s testing guidance recommends testing individual assets for fast feedback and using asset checks for runtime data-quality expectations. (https://docs.dagster.io/guides/test/unit-testing-assets-and-ops)
3. Asset-check tests — assert blocking checks fail on bad inputs and pass on good ones (red/green). (https://docs.dagster.io/guides/test/asset-checks)
4. Contract tests — SQL q_* shape/comments vs generated q_catalog; required provenance coverage vs q_provenance.
5. Integration/graph smoke — load the full Definitions and run a thin end-to-end slice.
6. Blank-database acceptance — materialize every asset and attached check from an empty PostgreSQL database, then execute P01-P12 as read-only SQL over the completed fixture graph.
7. Publication failure injection — for each generation-protected family, first serve a valid generation, force deterministic candidate and validator failures, prove the head and every stable model remain unchanged, then promote a second generation and atomically roll back the first.
8. Production packaging/topology acceptance — build locked gateway/web and Dagster images, validate installed definitions, boot the complete Compose stack against a blank disposable PostgreSQL volume, prove schema install/no-op, service health/auth denial, exact trusted GraphQL document enforcement, daemon/schedule state, non-root runtime, and clean teardown.

tests/test_query_publication.py is the reference failure-injection proof. It also proves promotion/rollback retry idempotence, physical generation immutability, contract-compatible rollback, and duplicate candidate-primary-key rejection before promotion. tests/test_slice_trials.py::test_db_smoke_full_chain exercises the same publisher with all seventeen real trial SELECTs and provenance/owner/domain checks, then injects an invalid arm-link state and proves the prior generation remains served.

flowchart BT
  Unit[Unit tests\nparsers, normalizers, merge logic]
  Asset[Asset tests\nin-process materialization]
  Checks[Asset-check tests\nblocking and warning gates]
  Contracts[SQL contract tests\nq_catalog + q_provenance]
  Smoke[Graph smoke tests\nDefinitions load + representative slices]
  Bootstrap[Blank-database acceptance\nall assets + checks + P01-P12, fixture-only]

  Unit --> Asset --> Checks --> Contracts --> Smoke --> Bootstrap

Blank-database acceptance gate

make verify-bootstrap is the completeness proof for graph ownership and bootstrap order. It:

1. creates a uniquely named disposable database on the local Compose PostgreSQL server;
2. removes LLM provider keys and BIOLOUPE_DATALAKE_URL from the materialization environment;
3. rejects any source resource whose default mode has drifted from checked-in fixtures;
4. materializes every loaded asset and every attached asset check in one Dagster run;
5. requires the terminal q_catalog checks to pass; and
6. forces the completed database connection read-only and executes all checked-in P01-P12 probes;
7. reports scenario-local evidence for failures; and
8. drops the disposable database even after failure.

Static ownership tests reject two Python owners for one literal DDL file and reject repositories that initialize another layer’s tables. The blank-database gate catches the remaining hidden dependencies and ordering defects: a query check reading an undeclared owner model, a downstream asset assuming an absent provenance relation, or a view depending on a relation that only happened to exist in a developer database. Use --keep-database through python -m bioloupe_data.lib.ops.bootstrap only when a failed run needs SQL inspection.

Agentic curation invariants (decided — must have tests)

From 05-agentic-curation-and-overrides.md:

• duplicate agent retry inserts one decision only (idempotency_key);
• supersession chain yields exactly one active effective assertion;
• high-impact pending proposal does not publish before human review;
• human-approved override outranks agent-only low-risk overlay (precedence);
• human-rejected proposal leaves previous effective assertion unchanged;
• sensor cursor + run_key prevent duplicate downstream runs;
• replaying the decision log rebuilds curation.effective_overrides deterministically.

Inference structuring/enrichment invariants (decided — must have tests)

From 06-llm-extraction-and-verification.md:

• one exact input + task contract revision produces the same provider-independent request key;
• provider/model/settings/transport changes produce a new attempt, not a new request;
• invalid enum, missing field, sentinel value, or source-quote mismatch produces findings and a rejected decision;
• only an accepted decision under the active acceptance-policy hash is reusable;
• a rejected output is never returned from InferenceRun.accepted and never reused;
• accepted evaluation attempts, including repeated samples, never become production cache hits;
• prompt/schema/source changes create a new request; validator/policy changes cannot reuse an old-policy decision;
• verifier rejection blocks high-impact canonical/query publication without becoming curation;
• batch submit/poll/ingest resumes across Dagster runs and matches results by custom_id, not order;
• a durable batch claim prevents concurrent duplicate submission and survives a fresh runtime on Dagster retry, while role/sample identity keeps deliberate cohorts distinct;
• batch cohort identity is independent of input order, and non-discounted batch capability is not confused with dispatch policy;
• in-flight batches use their persisted contract revision and execution-profile snapshot after route retuning;
• source-structured fields outrank inferred enrichment, and external calls happen outside domain write transactions.
• reviewed pricing is selected by provider/model pair, and an unknown pair fails before dispatch.
• missing/unknown or incomplete provider finish reasons reject before projection;
• cited artifact ids must be declared by the request, and claim text/numbers must occur in the same exact quote where the task requires it;
• default cohort caps fit their default hard budgets at maximum profile prompt/output size including every synchronous retry;
• profile-owned generation timeouts reach provider adapters, and an exhausted synchronous attempt persists the actual transport retry count;
• the revision-pinned semantic corpus covers every active task plus positive, negative, injection, numeric, identity-ambiguity, fill-missing, and regimen-confounder cases;
• offline goldens prove governance/scoring only; a paid provider-backed corpus run is required before route activation.

Definition-load test (decided)

dg check defs (and a pytest that imports the top-level Definitions) must pass in CI so a broken code location fails fast. (https://github.com/dagster-io/dagster/discussions/28472)

Source-resource testing contract (decided)

Every live-capable source family must prove the source seam offline. Dagster’s testing docs support direct asset invocation/materialization with mocked resources, and configurable resources can be constructed directly in tests. (https://docs.dagster.io/guides/test/unit-testing-assets-and-ops, https://docs.dagster.io/guides/build/external-resources/testing-configurable-resources)

Required tests for each source family:

1. The source-specific ConfigurableResource constructs in fixture mode without network access and returns typed capture records.
2. The same resource can be configured in live mode with a mocked source_http dependency; tests exercise the live adapter path without touching the internet.
3. The Dagster asset materializes with default resources in fixture mode.
4. A focused materialization/run-config test can switch the resource mode and max_live_records without changing asset code.
5. Materialization metadata reports source_mode, capture_scope, max_live_records when bounded, is_complete_source_capture, row counts, source release/version, and hash summary.
6. Default make check and CI never require live APIs, paid providers, or source credentials.
7. Every fixture-capable source resource rejects fixture mode when BIOLOUPE_RUNTIME_ENV=staging|production; an unknown tier fails rather than weakening policy.

Acceptance guard for touched source families: no asset-body BIOLOUPE_LIVE_SOURCE_* branch, no source-specific os.getenv mode parsing in defs/, and no direct source_http asset argument when a source-specific resource owns the family. Existing families that still violate this are migration targets, not patterns to copy.

Fixture evidence classes

lib/sources/fixture_manifest.yaml classifies every checked-in source fixture as an authored domain scenario, source-shaped test data, captured response excerpt, mixed real/authored data, or a project-owned catalog. Every entry has a review note and explicit live_parity_proof; current fixtures all declare false. tests/test_fixture_manifest.py rejects unregistered fixture files, unknown classes, missing notes, and accidental parity claims.

Fixtures prove deterministic behavior, SQL contracts, graph composition, and selected failure cases. They do not prove current upstream schemas, complete paging, source breadth, live parser behavior, or production quality. Those claims require separate bounded and complete live evidence under ADR-0012.

Slice Fixture Matrix

Slice family	Required fixture shape
Clinical trials	NCT/ChiCTR trial with arms, intervention synonym, disease, biomarker, endpoint, outcome, curation override
Regulatory approvals	FDA-like approval + label indication + disease/biomarker resolution + Orange Book-like product protection + immutable provenance bundles
Publications	PubMed-like abstract with NCT id, intervention, subgroup, endpoint, outcome statistic, abstract-only/full-text flag
News intelligence	press release with event taxonomy, org/drug/trial mentions, announcement-vs-confirmed semantics
Standard of care	authored regulatory approved-use candidate -> explicit non-SOC review row -> no automatic recommendation; separate human-approved promotion fixture
Organisation intelligence	org alias/name change/acquisition + temporal ownership interval + SEC/provider id
Discovery/search	projection-only boundary, canonical name lookup, real owner documents/chunks/mentions, owner-row provenance, entity-link scope, and SELECT-only recipe validation
Therapeutic development programs	same therapy in two diseases; terminated Phase 3 plus active Phase 2; approved plus active expansion; unknown line; source-specific registry/publication result conflict
Epidemiology	count/rate, crude/adjusted, population/geography/period, uncertainty, restatement, preferred-series selection, and projected-vs-observed cases
Commercial intelligence	Q4 plus annual values, reported/converted currency, geography split, restatement, ownership change, and prose-only non-numeric filing case
Forecasting/TPP boundary	optional readiness row + variable dictionary + assertion that no API/saved-model state enters core tables

Product Scenario Acceptance Pack

The product scenarios P01-P12 in 02-product-capability-contract.md are the replacement acceptance suite. The manifest and read-only runner live in src/bioloupe_data/lib/ops/product_scenarios.py; one checked-in SQL probe per scenario lives under src/bioloupe_data/sql/acceptance/product_scenarios/. make verify-bootstrap runs them only after the full fixture graph and every attached check have succeeded. The suite uses local Postgres and checked-in fixtures only: no live internet, source credentials, legacy database, or paid model call is allowed.

Scenario	Representative automated proof
P01 identity	aliases/external ids resolve to one owner key; ambiguous and type-conflicting mentions remain unresolved
P02 drug profile	one drug key joins brands, targets, approvals, trials, programs, commercial observations, ownership, and events through published models
P03 disease profile	one disease key joins taxonomy, biomarkers, epidemiology, SOC, approvals, programs, trials, and news without source-grain collapse
P04 development landscape	current/history/regulatory axes match program milestones and grouped counts reconcile to declared distinct keys
P05 trial detail	protocol, interventions, endpoints, eligibility, sites, sponsors, results, safety, and references retain source-native grains and provenance
P06 emerging evidence	registry and publication observations remain separate by arm/population/endpoint/timepoint and drill to exact source receipts
P07 approval vs SOC	regulator indication and non-SOC review candidate are visible while automatic label-to-SOC promotion remains empty
P08 organisation profile	temporal ownership, sponsorship, deal, filing, pipeline, marketed product, and reported-sales relations resolve as-of time
P09 disease burden	epidemiology filters return comparable contexts only and preferred summary values retain selection/source receipts
P10 commercial record	annual/quarterly/currency/geography/restatement/protection contexts remain distinct and no forecast masquerades as actual
P11 change/as-of	late source correction preserves observed/effective time and deterministically rebuilds historical state intervals
P12 find/explain	SQL-only recipe finds the row; q_catalog explains its columns; q_provenance reaches source and acceptance receipts

Each probe is a representative vertical proof over the shared fixture graph, not a claim of production source breadth. Adversarial red cases live beside the owning slice tests: ambiguity, wrong-domain matches, announcement-vs-effective dates, mixed statistic contexts, source-grain collapse, unresolved provenance, and unsupported summaries must fail before the scenario pack can be trusted. The scenario pack proves cross-slice composition; slice tests prove the important negative branches.

SQL Contract Testing Approach

Contract tests should be generated from the database-facing docs, not handwritten from memory.

Contract source	Test assertion	Failure meaning
q_catalog	every declared q_* relation/column exists and every physical column has description, grain, owner, version, enum, and provenance metadata	breaking query-layer drift
referential_integrity	every non-null cross-slice owner key (resolved_*_key(s), canonical_endpoint_key, segment_key, treatment_line_key) in a consumer q_* model matches an active row in its producing owner q_* model	dangling identity reference; orphan key namespace
q_provenance	every provenance-required published q_* row/fact has required receipts; bundle grain expands to source artifact, extraction run, evidence/canonical assertion, verifier/resolution receipt, and curation overlay only when applicable	untraceable or under-explained agent-visible fact
08 gate ids	representative bad fixture fails the correct blocking check	gate not enforced
slice fixture matrix	representative good fixture reaches expected q_* rows	asset graph integration broken
27 query recipes	recipe SQL parses and references existing read models only	docs teach invalid agent queries
28 boundary	grep/schema test proves no HTTP API/saved-model tables/contracts were added	downstream boundary violated
33 development program contract	program/context/state/milestone/evidence query models satisfy the orthogonal-state and temporal-grain contract	global status or hidden ranking logic reintroduced
34 epidemiology contract	observation/series/summary models preserve comparability dimensions and selection receipts	source statistic flattened into a disease attribute
35 commercial contract	sales/summary models preserve period/currency/geography/restatement and conversion receipts	incompatible commercial observations mixed

q_catalog contract fixtures

The catalog tests prove the checked-in registry, generated metadata asset, and Postgres physical catalog agree without inventing a second hand-maintained dictionary.

Required assertions:

1. Every registered contract column materializes once in ops.query_contract_columns.
2. Every physical q_* column maps to contract metadata or fails catalog_contract_coverage.
3. Small closed enums appear through allowed_values.
4. View nullability uses the checked-in contract because PostgreSQL reports view outputs nullable.
5. Schema or catalog drift fails before downstream publication.

Gate-factory red/green tests

Every reusable gate factory in 08-contracts-and-quality.md needs a red/green fixture pair:

tests/
  fixtures/
    gates/
      schema_contract/
        valid.sql
        drifted_missing_column.sql
      provenance_coverage/
        valid_bundle.sql
        orphan_q_row.sql
        missing_fact_path.sql
        partial_bundle_missing_decision.sql
      source_artifact_immutability/
        valid_archived_artifact.json
        missing_hash.json
        live_url_only.json
        changed_hash_same_snapshot.json
      source_authority/
        regulator_label_valid.json
        press_release_misused_as_approval.json
      development_program_semantics/
        active_phase2_after_stopped_phase3.json
        approval_ranked_as_phase_invalid.json
      epidemiology_observation_semantics/
        comparable_series_valid.json
        crude_adjusted_mixed_invalid.json
      commercial_observation_semantics/
        annual_and_q4_valid.json
        currency_overwrite_invalid.json

Minimum assertions:

• the valid fixture passes and emits expected metadata keys;
• the invalid fixture fails with the expected gate id;
• blocking factories are configured as blocking when attached to publication-path assets;
• warning factories record metadata without refreshing the gate vocabulary or silently passing as success;
• sample failing keys/locators are capped so check metadata is useful but not noisy.

This is the first implementable bridge between the docs and future code: the gate ids in 08, the asset keys in 04, and the q_* contracts in 07 all become executable assertions. The initial executable referential-integrity fixture now lives in tests/test_contracts.py: it proves a non-null canonical_endpoint_key passes when q_endpoints owns it, fails with capped metadata when dangling, and names the blocking referential_integrity__q_trial_outcomes check.

Inference fixture requirements

The inference test pack must prove that model calls are governed pipeline operators, not curation shortcuts:

tests/fixtures/llm/
  structured_output/
    valid_trial_endpoint.json
    invalid_enum_quarantined.json
    missing_source_locator.json
    sentinel_numeric_value.json
  enrichment/
    valid_enriched_field_with_basis.json
    enriched_field_missing_derivation_basis.json
    enriched_field_version_bump_new_run.json
    invalid_enriched_enum_quarantined.json
  idempotency/
    accepted_result_reused.json
    rejected_result_retried.json
    policy_revision_not_reused.json
    profile_change_new_attempt.json
  verification/
    producer_output_verifier_pass.json
    producer_output_verifier_reject.json
    ambiguous_candidate_quarantined_or_proposal.json
  provider_interface/
    openai_fake_sync_batch.json
    anthropic_fake_sync_batch.json
    together_fake_sync_batch.json
    gemini_fake_sync_batch.json
  batch_lifecycle/
    submit_handle_persisted.json
    poll_resume_complete.json
    ingest_results_by_custom_id.json

Required assertions:

1. Valid structured output creates evidence/resolution rows and extraction-run receipts.
2. Invalid structured output creates an attempt, findings, rejected decision, and no canonical/query publication.
3. Verifier rejection blocks high-impact publication but does not write curation.decision_events.
4. Reviewed curation overrides, when present, change effective canonical state without mutating raw LLM outputs.
5. Provider tests stay below the runtime; assets use task keys and InferenceInput, never vendor requests or SDKs.
6. Sync and batch paths produce the same attempt/finding/decision lifecycle.
7. Enriched fields carry an acceptance decision id and projection revision; deterministic source fields cannot be overwritten.
8. Reuse tests distinguish request identity, attempt identity, policy identity, and deliberate reuse_accepted=False evaluation attempts.
9. Inference fixtures must not write curation.decision_events; only a separate curator-agent/human-reviewed fixture exercises the curation overlay path.
10. Missing-target fixtures prove identity-gated nested citations: every artifact id exists in the input, every quote is an exact substring, and ambiguous identity forbids candidates.
11. Review-only task fixtures end in needs_review, write immutable suggestion/review receipts, and have no dependency path into canonical or query assets.
12. Every provider-calling asset defaults enabled=false, returns before database/runtime access when disabled, and enforces a finite cohort cap; default materialization therefore makes zero paid model calls even if an optional node is selected through downstream lineage.
13. At least one optional inference asset is exercised end to end against local Postgres with FakeProvider: request, attempt, findings, accepted decision, evidence projection, immutable projection receipt, accepted-result cache replay, and zero second provider calls.
14. Numeric eligibility fixtures prove scalar thresholds and two-bound ranges cannot be conflated; reversed or incomplete ranges are rejected before evidence projection.
15. SOC fixtures prove the deterministic claim asset has no LLM kind/resource, source values win over projections, accepted zero-field outputs receive replay receipts, and cache replay makes no second provider call.
16. FDA label fixtures are declared authored expectations rather than captured live responses; mocked live source_section rows remain deferred, bounded capture cannot move the active selector, application-scoped activation preserves unrelated labels, and no live section automatically enters canonical SOC.

Provenance fixture requirements

The provenance tests need red/green fixtures that prove the model is not just decorative:

Fixture	Expected result	What it protects
valid provenance bundle	pass	provenance_grain=bundle can expand from q_* row to source artifact, extraction run, evidence claim, verifier/resolution receipt, canonical assertion, and curation overlay only when applicable.
missing source hash	fail source_artifact_immutability	Live URLs or mutable payloads cannot masquerade as exact evidence.
missing required fact_path	fail provenance_coverage	Field/fact-level obligations are semantic and testable.
stale live URL without archived snapshot	fail source_artifact_immutability	Re-fetching a changed web/API payload does not rewrite history.
curation override bundle	pass and show reviewed-override precedence	Agent-proposed/human-reviewed curation overlays remain explainable after supersession.

These fixtures exercise the distinction in 07-sql-query-layer.md: prospective provenance is the q_catalog/gate expectation; retrospective provenance is the actual q_provenance receipt set emitted for a materialized row.

Comparative therapeutic-fact test pack

Add a small fixture pack that covers the first exact q_* contracts from 21/22/23:

tests/fixtures/comparative_facts/
  trial_registry_outcome/
    valid_orr_registry_bundle.sql
    missing_source_artifact.sql
    unaccepted_arm_link.sql
  regulatory_indication/
    valid_fda_label_indication.sql
    missing_region.sql
    translated_cde_missing_original_text.sql
  publication_outcome/
    valid_publication_hr_bundle.sql
    missing_text_locator.sql
    inferred_trial_link_without_verifier.sql
  cross_source_conflict/
    registry_vs_publication_different_orr.sql
    publication_misused_as_approval.sql

Required assertions:

1. Valid fixtures materialize their target q_* row and at least one complete q_provenance bundle.
2. Missing source artifact/hash/locator fixtures fail before query publication.
3. Publication-derived results never overwrite registry outcomes or label indications.
4. A registry-vs-publication conflict produces two provenance-backed facts with explicit roles, not a silent merge.
5. Inferred trial links and result-group-to-design-arm links require verifier acceptance or a reviewed curation override before downstream comparisons treat them as accepted.
6. q_catalog rows for the three query models expose nullability, owner, allowed values, provenance_required, and provenance_grain consistent with checked-in contracts.

Implementable red/green skeletons

Each fixture should be a minimal SQL/JSON bundle, not a full copy of production data. The fixture contract is: seed source artifacts + extraction/evidence/resolution/canonical rows, plus curation rows only when the fixture exercises an override, materialize the relevant query asset, run checks, then assert the expected query rows and provenance receipts.

Fixture	Seed minimal facts	Expected q_* row	Expected q_provenance receipts	Expected gate result
trial_registry_outcome/valid_orr_registry_bundle.sql	CTGov/AACT source artifact with hash; trial; endpoint ORR; source result group; accepted arm-link verifier/resolution receipt; outcome measurement count/percentage.	One q_trial_outcomes row with statistic_kind='count_of_participants' or percentage, arm_link_state='verifier_accepted', source_authority_tier='registry_result'.	Bundle contains source artifact, extraction run, evidence claim, arm-link verifier/resolution receipt, canonical trial outcome assertion, materialization run id.	pass schema_contract, provenance_coverage, source_artifact_immutability, resolution_basis, llm_verifier_required.
trial_registry_outcome/missing_source_artifact.sql	Same as valid, but no immutable source artifact hash/URI.	Candidate rejected and prior clinical-trials generation remains served.	Missing source artifact receipt.	fail source_artifact_immutability; fail or cascade-block provenance_coverage.
trial_registry_outcome/unaccepted_arm_link.sql	Outcome has result group label but no accepted design-arm link.	Row may publish only with design_arm_key is null and arm_link_state='unlinked'; no drug-arm comparison facts.	Bundle contains source artifact and outcome claim; no accepted arm-link verifier/resolution receipt.	pass as warning for row visibility; block if design_arm_key is non-null without accepted basis.
regulatory_indication/valid_fda_label_indication.sql	FDA/DailyMed label artifact with hash/version; approval assertion; indication text; accepted drug/disease/biomarker resolution links.	One q_approval_indications row with region='USA', approval_status='approved', source_authority_tier='regulator_label', raw indication text.	Bundle contains label artifact, extraction run, indication evidence claim, disease/drug resolution receipts, canonical indication assertion.	pass all blocking gates.
regulatory_indication/missing_region.sql	Approval/label evidence lacks region/regulator.	No published q_approval_indications row.	Source/evidence may exist, but canonical/query assertion is incomplete.	fail schema_contract/domain shape gate and source_authority.
regulatory_indication/translated_cde_missing_original_text.sql	CDE translated indication has English text but no original Chinese text or translation version.	No accepted translated indication row; may publish quarantined evidence only.	Missing original-language source locator or translation activity receipt.	fail document_version, source_locator, provenance_coverage.
publication_outcome/valid_publication_hr_bundle.sql	PubMed/PMC artifact with hash; publication-trial exact NCT link; endpoint PFS; HR/CI/p-value text locator.	One q_publication_outcomes row with trial_link_basis='explicit_registry_id', statistic_kind='hazard_ratio', evidence_coverage='full_text' or abstract_only.	Bundle contains publication artifact, extraction run, result evidence claim, trial-link resolution/verifier receipt, canonical publication outcome assertion.	pass all blocking gates.
publication_outcome/missing_text_locator.sql	Publication outcome statistic exists but lacks sentence/table/page/XML locator.	No published q_publication_outcomes row.	Source artifact may exist; fact-level locator receipt missing.	fail source_locator and provenance_coverage.
publication_outcome/inferred_trial_link_without_verifier.sql	Publication title/acronym suggests a trial but no registry id and no verifier acceptance or reviewed curation override.	Row may publish only as provisional evidence with trial_key is null or unresolved; no trial-level comparison.	Evidence receipt has trial_link_basis='title_acronym_match'; no accepted resolution/verifier receipt.	fail/block canonical comparison via resolution_basis/llm_verifier_required; query evidence may be warning/provisional.
cross_source_conflict/registry_vs_publication_different_orr.sql	Same trial/endpoint has registry ORR=28/84 and publication ORR=33/90.	Two rows remain in their own models; optional comparison view marks conflict, never overwrites.	q_trial_outcomes receipt role primary for registry fact; q_publication_outcomes receipt role supporting or conflicting for publication fact.	pass publication of both facts; warning/conflict check emits conflict metadata.
cross_source_conflict/publication_misused_as_approval.sql	Publication claims activity in disease; no regulator label/source.	No q_approval_indications row. Publication evidence may remain queryable in q_publication_outcomes/q_publication_evidence.	Publication provenance only; no regulator source authority receipt.	fail source_authority if any approval/indication assertion attempts to publish.

Publication Quality Regression Pack

The publication issue trackers become a first-class fixture pack, not a separate backlog. Each fixture is minimal and source-grounded: seed a publication artifact/abstract/table span, the LLM or parser output that would previously have gone wrong, canonical/resolution rows required for context, then assert the named gate from 08-contracts-and-quality.md fails or warns with diagnostic metadata.

tests/fixtures/publication_quality/
  value_semantics/
    zero_sentinel_orr_null_required.sql
    confirmed_orr_derive_from_cr_pr.sql
  endpoint_semantics/
    ttp_not_pfs.sql
    mean_survival_not_median.sql
    tumor_shrinkage_not_recist_orr.sql
    or_abbreviation_maps_to_orr_when_contextual.sql
  population_grain/
    parent_n_not_child_subgroup_n.sql
    biomarker_denominator_not_positive_subset.sql
    subgroup_disease_misattribution.sql
  arm_dose_attribution/
    all_arms_fanout_no_duplicate_dose.sql
    investigational_dose_not_control_arm.sql
    dose_escalation_not_rde.sql
  safety_attribution/
    cohort_specific_ae_not_all_arms.sql
    dose_arm_safety_not_sibling_arm.sql
  intervention_identity/
    non_drug_intervention_not_drug.sql
  corpus_identity/
    conference_and_journal_duplicate.sql
    pubmed_result_missing_from_corpus.sql

Fixture	Legacy issue evidence	Seed minimal facts	Expected gate result
zero_sentinel_orr_null_required.sql	Issue 8 / audit Finding 4	abstract has no ORR/N for subgroup but model output has 0; endpoint key ORR.	fail publication_value_semantics; no published numeric value.
confirmed_orr_derive_from_cr_pr.sql	Issue 54 / Issue 36	abstract reports cPR and CR=0 but not cORR; model emits cORR=0.	fail if cORR=0 is published; pass only if null or explicit derived ORR with derivation provenance.
ttp_not_pfs.sql	Issue 32	source phrase is TTP, candidate endpoint PFS.	fail endpoint_semantic_consistency; canonical_endpoint_key must not be PFS.
mean_survival_not_median.sql	Issue 37	source says mean survival; output statistic says median.	fail endpoint/statistic semantic consistency.
tumor_shrinkage_not_recist_orr.sql	Issue 42	tumor shrinkage rate extracted as ORR.	fail endpoint_semantic_consistency.
or_abbreviation_maps_to_orr_when_contextual.sql	Issue 61	source uses endpoint abbreviation OR in response context.	fail if invisible/unmapped; pass if mapped to ORR with contextual evidence.
parent_n_not_child_subgroup_n.sql	Issues 26 / 40	child subgroup missing N; parent population N copied down.	fail population_grain_integrity.
biomarker_denominator_not_positive_subset.sql	Issue 45	source says X/Y biomarker-positive/tested; subgroup N uses Y.	fail population_grain_integrity; subgroup N should be X or null if unsupported.
subgroup_disease_misattribution.sql	Issue 57	biomarker-named cohort assigned wrong disease.	fail population_grain_integrity / disease-resolution basis.
all_arms_fanout_no_duplicate_dose.sql	Issue 52	All Arms view joins to every intervention and duplicates conflicting doses.	fail arm_dose_attribution; no duplicate dose rows publish.
investigational_dose_not_control_arm.sql	Issue 31	investigational drug dose appears on control/comparator arm.	fail arm_dose_attribution.
dose_escalation_not_rde.sql	Issues 29 / 58	escalation range captured as recommended dose/RDE.	fail arm_dose_attribution.
cohort_specific_ae_not_all_arms.sql	Issue 53	AEs reported per cohort but flattened to all arms.	fail safety_attribution_integrity unless cohort linkage preserved.
dose_arm_safety_not_sibling_arm.sql	Issue 41	safety N/discontinuation from one dose arm attached to sibling arm.	fail safety_attribution_integrity.
non_drug_intervention_not_drug.sql	Issue 50	music/procedure/device/control term resolved to Drug.	fail intervention_identity_type_safety; route to NCIt/non-drug or unresolved.
conference_and_journal_duplicate.sql	Issue 17	ASCO/conference and journal article duplicate same result.	warn or block via publication_corpus_identity depending on duplicate query-row impact.
pubmed_result_missing_from_corpus.sql	Issue 18	PubMed-indexed result article omitted from corpus.	warning/coverage failure via publication_corpus_identity.

These fixtures are part of the completion bar for publication quality. A future build is not allowed to claim publication-result correctness by passing only generic schema/provenance tests; it must also pass this regression pack or explicitly mark unsupported fixture families.

Fixture assertion shape

A future builder can express each fixture test with the same assertion grammar:

Given fixture <name>
When materializing <asset selection>
Then expected_query_rows = <count and key predicates>
And expected_provenance_receipts include <receipt roles and ids>
And expected_gate_results include <gate_id, severity, pass/fail, failing keys>
And publication_protected = <true/false>

For example:

Given trial_registry_outcome/missing_source_artifact.sql
When materializing query.q_trial_outcomes
Then expected_query_rows = 0 new rows for trial_outcome_key X
And expected_gate_results include source_artifact_immutability:blocking:fail
And expected_gate_results include provenance_coverage:blocking:fail
And publication_protected = true

The example above is a required acceptance test for the future generation/promotion mechanism. Current check metadata emits publication_protected=false; changing the assertion to true without a checksum-preserving failure-injection test is forbidden.

CI commands (decided baseline)

uv sync --extra dev
make check
make verify-bootstrap

Locked Testing Defaults

• SQL contract tests use checked-in YAML interpreted against Postgres information_schema. The representation may evolve without changing that ownership boundary.
• Default inference fixtures store the provider-neutral validated request/result shape. Adapter tests may additionally keep tiny redacted provider payloads when a provider-specific quirk requires a regression fixture; raw production responses are not test fixtures.