23 Slice Publications

Pipeline Slice: Publications and Conference Evidence

Status: SHARP CORE VERTICAL. Fixture/live/import PubMed and Crossref capture, fixture/import conference capture, deterministic claims, conservative resolution, canonical publication evidence, six query models, checks, and provenance are implemented. PMC full-text breadth and evaluated table extraction remain bounded expansions. Denoising state: sharp (noise -> rough -> sharp -> converged) Updated: 2026-07-23

Purpose

Model how publications, abstracts, conference proceedings, and full-text artifacts become structured clinical evidence. This slice owns publication identity, publication-to-trial linking, abstract/full-text evidence claims, subgroup/endpoint/outcome extraction, and publication-derived evidence read models.

Legacy Data Gov treated publications as a separate 18-step LLM-heavy pipeline: identify trial result publications, extract/verify trial identifiers, extract interventions, subgroups, endpoints, outcomes, adverse events, classify purpose, and link to trial/disease/endpoint assets (../bioloupe-data-gov/docs/publications-workflow.md, ../bioloupe-data-gov/docs/publication-data-model.md, ../bioloupe-data-gov/app/workflows/publications_workflow.rb).

Current Decided Shape

Publications produce evidence claims and reported-result context, not canonical trial truth directly. A paper can strengthen, contradict, or extend registry/label data, but it must pass through provenance, entity resolution, verifier/check policy, and any applicable reviewed curation overlay before affecting canonical/query assets.

flowchart LR
  PubMed[PubMed / PMC]
  Crossref[Crossref / DOI metadata]
  Conference[Conference abstracts / PDFs]
  FullText[Full text / Unpaywall / publisher locators]
  Raw[Raw publication snapshots]
  Normalized[Normalized publication records]
  Evidence[Publication evidence claims]
  Resolution[Trial, drug, disease, endpoint, arm resolution]
  Verify[Verifier / accepted basis]
  Curation[Reviewed curation overlay]
  Canonical[Canonical publication evidence]
  Query["q_publication_evidence family"]

  PubMed --> Raw
  Crossref --> Raw
  Conference --> Raw
  FullText --> Raw
  Raw --> Normalized --> Evidence --> Resolution --> Verify --> Canonical --> Query
  Curation -. optional override .-> Canonical

Source Inventory

Source	Greenfield role	Source identifiers	Legacy evidence	Web/reference evidence
PubMed / NCBI E-utilities	biomedical citation and abstract source	PMID, PMCID when available	publications collected and linked via PMID/trial identifiers	NCBI E-utilities provide public API access to PubMed/PMC and other Entrez databases: https://www.ncbi.nlm.nih.gov/home/develop/api/
PubMed Central	open-access full text where available	PMCID	full text fetched when available in legacy publications collection	PMC is accessed through NCBI/Entrez tooling and full-text resources: https://pmc.ncbi.nlm.nih.gov/tools/id-converter-api and https://pmc.ncbi.nlm.nih.gov/tools/oai/
Crossref	DOI and scholarly metadata reconciliation	DOI	source metadata/DOI support in publication model	Crossref REST API exposes deposited scholarly metadata: https://www.crossref.org/documentation/retrieve-metadata/rest-api/
Unpaywall	open-access full-text locator and OA status	DOI	legacy notes full text fetched from PMC or Unpaywall when available (../bioloupe-data-gov/docs/news-and-intel.md)	Unpaywall provides a REST API requiring email parameter: https://unpaywall.org/products/api
Conference abstract sources	oncology/hematology conference abstract books and feeds	source + source_id / abstract number	ASCO/AACR/ASH/ASGCT/EHA/ESMO and conference_pdf ingestion described in legacy docs	Source-specific; preserve public URL/PDF/source locator.
ClinicalTrials.gov / canonical trials	trial identity and registry linkage target	NCT / other registry ids	publication_clinical_trials join table and trial-id extraction	Clinical-trials slice owns canonical trial identity; publication slice owns evidence-to-trial links.

Source Decision

PubMed/PMID is the first-class publication identity spine, DOI is a cross-source metadata key, and conference sources are first-class source families because oncology evidence often appears first as abstracts. Full text is opportunistic evidence, not a requirement for ingestion. Abstract-only records remain usable with lower evidence coverage.

Asset Graph for This Slice

flowchart TD
  Sources[[Publication sources]]
  Raw[sources.publications.*_snapshots]
  Normalized[normalized.publication_records]
  Identity[evidence.publication_identity_claims]
  TrialClaims[evidence.publication_trial_link_claims]
  InterventionClaims[evidence.publication_intervention_claims]
  ResultClaims[evidence.publication_result_claims]
  SafetyClaims[evidence.publication_safety_claims]
  Resolution[resolution.publication_entities]
  Curation[[curation.effective_overrides]]
  Canonical[canonical.publication_evidence]
  Query["q_publication_evidence family"]

  Sources --> Raw --> Normalized --> Identity --> Resolution
  Normalized --> TrialClaims --> Resolution
  Normalized --> InterventionClaims --> Resolution
  Normalized --> ResultClaims --> Resolution
  Normalized --> SafetyClaims --> Resolution
  Resolution --> Canonical --> Query
  Curation -. reviewed overrides .-> Canonical

Slice-Local Domain Model

Publications stress reported evidence, subgroup semantics, trial linkage, and abstract-vs-full-text evidence quality.

Entity	Grain	Canonical key recommendation	Notes
Publication source record	one source publication/abstract record	publication:{source}:{source_id}	PMID, DOI, conference abstract id are source-scoped.
Publication version/snapshot	one retrieved source payload/time	publication_snapshot:{source}:{source_id}:{seen_at_or_version}	Abstracts and metadata can update.
Publication-trial link	one publication-to-trial claim	publication_trial_link:{publication_key}:{trial_key_or_registry_term}	Link can be exact registry id or inferred.
Publication intervention	one intervention/treatment arm mention	publication_intervention:{publication_key}:{mention_or_arm_hash}	May represent study-level, subgroup-level, or arm-level treatment context.
Publication subgroup	one reported population/cohort	publication_subgroup:{publication_key}:{subgroup_hash}	Disease, biomarker, dose, prior therapy, line, geography, age, performance-status dimensions.
Publication endpoint	one endpoint/measure definition	publication_endpoint:{publication_key}:{endpoint_hash}	Endpoint terms are evidence until linked to canonical endpoint catalog.
Publication outcome measure	one endpoint × subgroup/statistic claim	publication_outcome:{publication_key}:{subgroup}:{endpoint}:{stat_hash}	Holds ORR/PFS/OS/HR/p-value/CI/etc with source text.
Publication arm outcome	one arm-level result under an outcome	publication_arm_outcome:{outcome_key}:{arm_hash}	Arm may later link to registry design arm.
Publication adverse event claim	one safety event/result claim	publication_ae:{publication_key}:{event}:{arm_or_group_hash}	Can feed safety evidence but not label truth.

Invariants

• A publication evidence claim must retain source text span or source locator.
• A publication-to-trial link based on explicit registry id is stronger than inferred title/acronym matching.
• Subgroup and arm are different grains. A subgroup may be disease/biomarker/dose/population; an arm is an intervention allocation/result group.
• Abstract-only extraction is allowed but must expose evidence coverage so agents know whether full text was available.
• Publication-derived results do not overwrite registry/label facts; they become provenance-backed evidence assertions that can be queried and compared.
• Raw publication versions remain append-only. The normalized source record selects exactly one current version per PMID, DOI, or conference abstract by seen_at, then fetched_at, then snapshot_id; it never relies on database fetch order.
• The selected normalized row cites the exact content-addressed artifact key containing the full payload hash, not a truncated digest or merely the newest artifact for that source id.

Slice-Local Entity Resolution

Subject kind	Source term examples	Candidate target types	Publishable basis	High-impact?
publication_trial_link	NCT ID, trial acronym, title fragment, EudraCT/ISRCTN	canonical Trial	exact registry id, source cross-reference, verifier or reviewed curation override for inferred	yes
publication_intervention_link	pembrolizumab, arm A, dose cohort	drug spine resolved_drug_key / NCIt non-drug concept / regimen component / trial arm	exact synonym, context-verified, verifier-passed LLM, reviewed curation override	yes
publication_disease_link	disease populations, histology subgroup	canonical Disease / NCIt	exact/NCIt, semantic+verifier, reviewed curation override	yes
publication_biomarker_link	PD-L1 TPS, EGFR mutation, HER2+	canonical Biomarker	exact/cross-reference/context-verified/reviewed curation override	yes
publication_endpoint_link	ORR, cORR, PFS, OS, DOR, MRD	canonical Endpoint catalog	exact abbreviation map, accepted synonym, verifier or reviewed curation override	medium/high
publication_arm_to_trial_arm_link	publication arm name vs registry study plan arm	canonical trial design arm	exact source id if present, verifier or reviewed curation override otherwise	high

Evidence Claim Schemas

publication_identity_claim

Field	Meaning
publication_key	source-scoped publication id
source_family	PubMed / PMC / Crossref / conference / publisher / other
pmid / pmcid / doi	external ids where present
title / abstract / journal_or_meeting	source metadata
publication_date	date + precision
source_locator	API URL/PDF/page locator

publication_trial_link_claim

Field	Meaning
publication_key	publication id
registry_id_term	raw NCT/EudraCT/ISRCTN/etc term if present
trial_title_or_acronym	raw trial reference text
link_basis	explicit_registry_id / reference_section / title_match / web_search / inferred
candidate_trial_keys	candidate canonical trial keys
evidence_text	source span

publication_result_claim

Field	Meaning
publication_key	publication id
source_artifact_key	immutable abstract/full-text/PDF/table source snapshot key
source_locator	abstract sentence, full-text section/table/page/span, XML path, or PDF locator
subgroup_term	raw subgroup/cohort term
endpoint_term	raw endpoint/measure term
arm_term	raw treatment arm/group term
statistic	value, unit, CI, p-value, HR/OR/RR, time point
population_n	denominator if extractable
evidence_text	source sentence/table/span
resolution_subject_keys	trial/intervention/disease/endpoint/arm subjects

Curation Override Subjects

Subject kind	Why overrideable	Example curation decision
publication_trial_link	Incorrect trial links corrupt evidence attribution.	PubMed article X reports NCT Y, not similarly titled NCT Z.
publication_intervention_link	Treatment names in abstracts are noisy and context-dependent.	“chemo” remains regimen concept; pembrolizumab resolves to Drug X.
publication_subgroup_assertion	Subgroups define evidence applicability.	“EGFR-mutant NSCLC” is disease + biomarker subgroup, not a separate disease alone.
publication_endpoint_link	Endpoint normalization controls efficacy comparisons.	cORR maps to ORR with confirmed flag.
publication_arm_to_trial_arm_link	Arm result values need correct registry/label arm context.	“8 mg/kg cohort” links to registry arm A.
publication_result_assertion	Numeric outcomes can be misread from abstracts/tables.	ORR=33.3% for subgroup S and arm A, source text span T.

Canonical Assets

Canonical asset	Grain	Built from
canonical.publications	one publication identity	publication snapshots + identity resolution/verifier receipts; curation overlay when present
canonical.publication_trial_links	one accepted publication-trial relation	trial link claims + resolution/verifier receipts; curation overlay when present
canonical.publication_interventions	one accepted intervention mention/link	intervention claims + resolution
canonical.publication_subgroups	one population/cohort assertion	subgroup claims + disease/biomarker resolution/verifier receipts; curation overlay when present
canonical.publication_endpoints	one endpoint assertion	endpoint claims + endpoint resolution/verifier receipts; curation overlay when present
canonical.publication_outcomes	one reported result statistic	result claims + verifier/check receipts; curation overlay when present
canonical.publication_adverse_events	one safety claim	AE claims + verifier/check receipts; curation overlay when present

q_publication_evidence Family

Query model	Grain	Purpose
q_publications	one publication	citation/abstract search and source coverage
q_publication_trial_links	one publication-trial link	evidence attribution to trials
q_publication_evidence	one evidence claim with accepted resolution/verifier state	broad evidence search by drug/disease/endpoint/trial
q_publication_outcomes	one endpoint/group/statistic	efficacy/safety result querying
q_publication_subgroups	one subgroup/cohort	population applicability and biomarker/disease cuts
q_publication_adverse_events	one AE/safety claim	safety evidence exploration

q_publication_outcomes comparative fact contract

q_publication_outcomes is the publication-evidence side of the shared therapeutic-fact model in 20-pipeline-families.md. It is a reported-result claim from an abstract/full-text/table, not a registry result row and not an approved indication. PubMed/PMC provide citation, abstract, and full-text artifact structures through NCBI tooling; Crossref/DOI and Unpaywall help find metadata and open full text, but the evidence row must still preserve the exact text/table locator used for extraction (NCBI E-utilities: https://www.ncbi.nlm.nih.gov/home/develop/api/; PubMed XML elements: https://www.nlm.nih.gov/bsd/licensee/data_elements_doc.html; PMC OAI/ID tools: https://pmc.ncbi.nlm.nih.gov/tools/oai/ and https://pmc.ncbi.nlm.nih.gov/tools/id-converter-api/).

Contract field	Required meaning
Purpose	Answer: “What did this publication report for this trial/subgroup/arm/endpoint, and how strong is the evidence coverage?”
Grain	One extracted reported-result statistic or explicit non-numeric result for one publication + trial/link context + endpoint + subgroup/population + arm/group + statistic kind.
Stable row key	publication_outcome:{publication_key}:{trial_link_key}:{endpoint_hash}:{subgroup_hash}:{arm_hash}:{statistic_kind}.
Required columns	publication_outcome_key, publication_key, pmid, doi, trial_key, trial_link_basis, endpoint_term, canonical_endpoint_key, subgroup_label, disease_terms, resolved_disease_keys, biomarker_terms, resolved_biomarker_keys, arm_label, resolved_drug_keys, population_n, statistic_kind, statistic_value, statistic_unit, hazard_ratio, odds_ratio, risk_ratio, ci_lower, ci_upper, p_value, time_point, evidence_coverage, source_artifact_key, source_locator, assertion_state, provenance_key.
Search affordances	filters on PMID/DOI/trial, endpoint, disease, biomarker, drug/intervention, evidence coverage, statistic kind; full-text/trigram over title/abstract/arm/subgroup/endpoint terms.
Provenance rule	provenance_grain=bundle for every row; field receipts for numeric statistics, confidence intervals, p-values, trial_link_basis, and evidence_coverage.
Refresh/materialization	current runtime is an in-place plain view after publication identity, trial-link, endpoint/intervention/disease resolution, result verifier gates, and applicable curation-review gates; it is not last-known-good protected until the publications family adopts atomic generation publication.
Blocking checks	schema_contract, provenance_coverage, source_artifact_immutability, source_locator, resolution_basis, llm_verifier_required, curation_review_state when override applies, announcement_semantics only when the publication row is an event/announcement rather than a result.

Allowed fact_path examples:

publication_outcome:pmid=123:trial=NCT01234567:endpoint=pfs:arm=control:attr=hazard_ratio
publication_outcome:pmid=123:trial=NCT01234567:endpoint=orr:subgroup=egfr_mutant:attr=statistic_value
publication_outcome:pmid=123:trial=NCT01234567:attr=trial_link_basis

Registry q_trial_outcomes can be compared with this row, but comparison is an explicit synthesis step. The publication row remains source-authored evidence, with evidence coverage (abstract_only, full_text, table_extracted, etc.) visible to agents.

q_publication_outcomes column contract seed

Column	Type	Nullability	Semantic type	Provenance
publication_outcome_key	text	not null	query row key	row
publication_key	text	not null	publication key	row
pmid	text	nullable	PMID	row when present
doi	text	nullable	DOI	row when present
trial_key	text	nullable	linked canonical trial key	field when present
trial_link_basis	text	not null	trial-link basis	field
endpoint_term	text	not null	source endpoint phrase	bundle
canonical_endpoint_key	text	nullable	resolved endpoint key	field when present
subgroup_label	text	nullable	source subgroup/cohort	bundle when present
disease_terms	text[]	nullable	source disease phrases	bundle when present
resolved_disease_keys	text[]	nullable	resolved disease keys	field when present
biomarker_terms	text[]	nullable	source biomarker phrases	bundle when present
resolved_biomarker_keys	text[]	nullable	resolved biomarker keys	field when present
arm_label	text	nullable	source arm/group label	bundle when present
resolved_drug_keys	text[]	nullable	resolved intervention/drug keys	field when present
population_n	numeric	nullable	denominator/count	field when present
statistic_kind	text	not null	statistic kind	bundle
statistic_value	numeric	nullable	numeric result	field when present
statistic_unit	text	nullable	unit	field when present
hazard_ratio	numeric	nullable	hazard ratio	field when present
odds_ratio	numeric	nullable	odds ratio	field when present
risk_ratio	numeric	nullable	risk ratio	field when present
ci_lower	numeric	nullable	confidence interval lower	field when present
ci_upper	numeric	nullable	confidence interval upper	field when present
p_value	numeric	nullable	p-value	field when present
time_point	text	nullable	reported time point	bundle when present
evidence_coverage	text	not null	abstract/full-text/table coverage	field
source_artifact_key	text	not null	immutable source artifact	bundle
source_locator	jsonb	not null	source locator	bundle
source_authority_tier	text	not null	authority tier	row
assertion_state	text	not null	effective assertion state; curation overlay if applicable	row
provenance_key	uuid	not null	provenance bundle	bundle

Catalog notes: trial_key may be null when a paper reports a relevant result but the trial link is unresolved; in that state the row may appear only as provisional/searchable evidence and must not feed trial-level canonical comparisons. evidence_coverage should be a controlled vocabulary such as abstract_only, full_text, table_extracted, poster_pdf, or metadata_only.

q_publication_outcomes vocab/search/example seed

Legacy data-shape pressure from local datalake confirms publication sources are heterogeneous: publications.source includes PubMed plus conference/import families such as AACR, ASCO, ESMO, EHA, ASH, IJGC, file imports, and news-linked rows; full text is sparse and currently PMC-shaped where present (full_text_source=pmc, full_text_format=html in the sampled stats). This supports keeping evidence_coverage explicit instead of pretending every publication row has full-text/table evidence (public.publications, sampled from local datalake).

Controlled vocabularies:

Field	Controlled values / normalization
source_family / source	pubmed, pmc, crossref, asco, aacr, ash, eha, esmo, ijgc, publisher, conference_pdf, news_linked, manual_import.
trial_link_basis	explicit_registry_id, publication_metadata, reference_section, title_acronym_match, verifier_inferred, curation_override, unresolved.
statistic_kind	orr, cr, pr, dcr, pfs, os, dor, hazard_ratio, odds_ratio, risk_ratio, count, percentage, median, p_value, adverse_event_rate, textual.
evidence_coverage	metadata_only, abstract_only, full_text, table_extracted, poster_pdf, conference_abstract, publisher_locator_only.
source_authority_tier	peer_reviewed_publication, conference_abstract, preprint_or_poster, publication_metadata, news_derived_publication_signal.
assertion_state	accepted, agent_verified, curation_override, provisional, conflicting, superseded, rejected.

Search/index contract:

Search need	Required index/affordance
Find evidence by publication id	btree on pmid, doi, and publication_key.
Trial evidence lookup	btree on trial_key; partial index where trial_key is null for unresolved evidence.
Endpoint/result search	btree on canonical_endpoint_key, statistic_kind; trigram/full-text on endpoint_term, arm_label, subgroup_label.
Drug/disease/biomarker cuts	GIN on resolved_drug_keys, resolved_disease_keys, and resolved_biomarker_keys.
Evidence quality filtering	btree on evidence_coverage, source_authority_tier, and assertion_state.
Numeric comparisons	partial btree indexes on hazard_ratio, odds_ratio, risk_ratio, statistic_value, and p_value where not null.

Example row shape:

publication_outcome_key: publication_outcome:pmid-123:NCT01234567:pfs:control:hazard_ratio
publication_key: publication:pubmed:123
pmid: 123
trial_key: trial:ctgov:NCT01234567
trial_link_basis: explicit_registry_id
endpoint_term: progression-free survival
canonical_endpoint_key: endpoint:pfs
subgroup_label: PD-L1 positive population
arm_label: control arm
resolved_drug_keys: [drug:docetaxel]
statistic_kind: hazard_ratio
hazard_ratio: 0.72
ci_lower: 0.55
ci_upper: 0.94
p_value: 0.01
evidence_coverage: full_text
source_artifact_key: publication_artifact:pmc:PMC123:v1
source_authority_tier: peer_reviewed_publication
provenance_key: <uuid>

Pathological examples the implementation must handle:

Case	Why it is dangerous	Required behavior
Trial link is inferred from title/acronym only	Wrong trial attribution corrupts every cross-source comparison.	Publish only provisional/searchable evidence with trial_key is null or unresolved until verifier or reviewed curation override acceptance.
Abstract reports pooled “All Arms” result while arms have different doses	Legacy publication issues showed broad “All Arms” joins can fan out dose/drug attribution.	Keep arm_label='All Arms'/pooled context; do not infer per-arm drug/dose attribution without accepted arm/subgroup link.
Full text unavailable but abstract has a statistic	Abstract-only evidence is useful but lower coverage.	Publish with evidence_coverage='abstract_only', source locator to abstract sentence, and lower source authority than full text/table extraction.
Background section cites prior-trial result	LLM extraction can confuse cited background facts with results of the current paper.	Require source section/table locator and claim role; background-cited facts must not become accepted current-publication outcomes without verifier acceptance.

Checks and Quality Gates

Blocking:

• every published evidence row has source locator/text span;
• every exact registry-id publication-trial link resolves to an existing canonical trial or remains unpublished as unresolved;
• every resolved_drug_keys, resolved_disease_keys, resolved_biomarker_keys, canonical_endpoint_key, and resolved_trial_key (via trial_key) value resolves to an active row in its owner q_* model (referential_integrity gate in 08-contracts-and-quality.md);
• no inferred publication-trial link publishes without verifier or reviewed curation override acceptance;
• every outcome row has endpoint term, statistic value or explicit non-numeric value, and subgroup/arm context where present;
• schema of q_publication_* matches generated q_catalog;
• provenance-required query rows/facts resolve through q_provenance.

Warning:

• abstract-only evidence coverage;
• full-text fetch failure;
• high unmatched intervention/disease/endpoint rate;
• duplicate DOI/PMID/source id conflicts;
• unusually high LLM extraction failure/empty-output rate.

Source Expansion Boundary

• Add source-specific conference collectors only where access, release identity, and replay semantics are explicit; generic live scraping is not a source contract.
• Expand PMC/full-text capture and evaluated table extraction behind the same artifact and locator contracts.
• Keep typed outcome/subgroup/adverse-event models as owners and q_publication_evidence as their agent-facing union; do not replace them with one write-time wide table.
• Publication-derived adverse events stay publication evidence and resolve through the shared AE vocabulary; a future safety synthesis may consume them without taking ownership.