29 Slice Drug Identity
Pipeline Slice: Drug / Therapeutic Product / Intervention Identity Spine
Section titled “Pipeline Slice: Drug / Therapeutic Product / Intervention Identity Spine”Status: SHARP FOUNDATIONAL VERTICAL. Regulatory active-substance/brand claims, cache-first NCIt/RxNorm enrichment, canonical identities, aliases, external ids, brands, regimens, shared name resolution, five query models, checks, provenance, and legacy reconciliation sidecars are implemented. Contextual development state is deliberately absent from this spine. Denoising state: sharp (noise -> rough -> sharp -> converged) Updated: 2026-07-23
Purpose
Section titled “Purpose”Define the shared drug / therapeutic product / intervention identity spine that every other slice resolves into. In legacy Data Gov, drugs are explicitly the center of the knowledge graph: drugs are tracked at INN/compound level, the same compound appears under many names (“Pembrolizumab”, “KEYTRUDA”, “MK-3475”, “lambrolizumab”), and trials, approvals, publications, news, SOC, organisations, and forecasting all hang off drug identity (../bioloupe-data-gov/docs/index.md, ../bioloupe-data-gov/docs/data-model.md “Domain 1: Drugs”). This slice owns that identity so downstream resolved_drug_key(s) references have a real producer.
This spine deliberately separates grains that legacy collapsed into one drugs table plus satellites: active substance/ingredient, branded/marketed product, regulatory product/application, combination/regimen, trial/publication intervention component, and non-drug intervention. Flattening these is the documented source of matching bugs (../bioloupe-data-gov/app/services/drug_matching_service.rb formulation/isotope guards; ../bioloupe-data-gov/docs/publication_issues_tracker.md issue 50 “DrugLinker false-matches non-drug interventions to drugs”). It also separates stable identity from contextual development state: phase, clinical activity, regulatory state, and historical progression belong to therapeutic development programs in 33-slice-development-landscapes.md, not to a drug row.
Current Decided Shape
Section titled “Current Decided Shape”Drug identity is a foundational spine, not a downstream slice. It produces canonical drug nouns and the resolution surface that trials/approvals/publications/news/SOC/organisations consume. It does not own approvals, trial design, therapeutic development state, or SOC assertions — only identity and the links into it.
flowchart LR Sources["source artifacts\n+ authority vocabularies"] Evidence["drug identity / alias / product claims"] Resolution["intervention resolution candidates + verifier"] Canonical["canonical.drugs + ingredients + products + regimens + external ids"] Curation["reviewed merge/split/identity overrides"] Query["q_drugs family"] Sources --> Evidence --> Resolution --> Canonical --> Query Curation -. correction / merge / split .-> Canonical
Source Inventory
Section titled “Source Inventory”| Source | Access | Identifier | Role |
|---|---|---|---|
| ChEMBL (project-deprecated legacy remainder) | mapping-only if salvaged | chembl_id | Deprecated for this greenfield repo, not modelled as a live ingestion source. Legacy carries ~3.7k source_drugs.source='chembl' rows and drugs.chembl_id, but the product decision is to stop carrying ChEMBL as a go-forward source. Upstream ChEMBL itself remains active (EMBL-EBI describes ChEMBL as a current curated bioactivity database and published ChEMBL 36 as live in 2025: https://www.ebi.ac.uk/chembldb/ and https://www.ebi.ac.uk/about/news/updates-from-data-resources/chembl-36/). Therefore greenfield may retain old ChEMBL ids only as deprecated historical mappings; do not add new ChEMBL ingestion. |
| NCI Thesaurus (NCIt) | reference vocabulary | ncit_id / NCIt code | drug and non-drug concept vocabulary; NCI describes NCIt as its core reference terminology and biomedical ontology (https://www.cancer.gov/about-nci/organization/cbiit/vocabulary). Procedures/devices/radiation live here (../bioloupe-data-gov/app/models/source_drug.rb, ncit_concepts). |
| RxNorm | reference vocabulary | rx_norm_id | normalized clinical-drug naming; NLM says RxNorm provides normalized names for clinical drugs and links to common drug vocabularies (https://www.nlm.nih.gov/research/umls/rxnorm/index.html). Legacy source_drugs rows use source='rxnorm'. |
| Adis | legacy/proprietary mapping unless source contract is re-established | adis_id | Development-stage drug naming/synonyms (source_drugs.source='adis') from a paid/proprietary source family, not a public source-first adapter. Springer describes AdisInsight as a drug R&D intelligence database with platform/API/data-feed access (https://adisinsight.springer.com/). Salvage existing ids/names as mapping-only traceability; do not mint canonical drugs from Adis unless a current source contract, release/version, and ingestion adapter are supplied. |
| Regulatory products | from approvals slice | brand/application/ingredient | brand/product identity evidence (brand_drugs, FDA/EMA/KEGG/CDE); FDA Orange Book data files identify application products (https://www.fda.gov/drugs/drug-approvals-and-databases/orange-book-data-files) and Purple Book lists approved biological products, biosimilars/interchangeables, and reference products (https://purplebooksearch.fda.gov/). |
| Free-text mentions | trials/publications/news | intervention strings | identity demand, resolved into this spine, not new identity. |
Live shape sampled from datalake (2026-06): active drugs 13,967, active brand_drugs 13,118, active source_drugs 19,237 (Adis 6,909, NCIt 6,589, ChEMBL 3,736, RxNorm 2,003), trial_arm_interventions≈95.2k (each carrying drug_id xor ncit_concept_id), publication_interventions≈60.1k (drug-linked ≈39.6k, ncit-linked ≈47.6k). This confirms two resolution targets — canonical drug and NCIt concept — not one.
Source liveness decision. Presence of source_drugs rows is not proof of an active source. A 2026-06 read-only legacy sample shows source_drugs is load-bearing as a historical cross-reference table, but not clean evidence that every source should be re-ingested: active source_drugs counts were Adis 6,909, NCIt 6,589, ChEMBL 3,736, RxNorm 2,003; linked active drugs were Adis 6,739, NCIt 6,447, ChEMBL 3,601, RxNorm 1,940; 5,506 active drugs had no source-vocabulary FK at all. The liveness split is:
| Source | Legacy DB signal | Runtime signal | Greenfield decision |
|---|---|---|---|
| NCIt | recent updates through 2026-06-18; 6,447 linked active drugs | active NCIt concept linking/conversion in ../bioloupe-data-gov/app/models/ncit_concept.rb and ../bioloupe-data-gov/app/admin/ncit_concepts.rb | live reference vocabulary; the implemented path enriches an existing regulator-derived identity and routes collisions to review. |
| RxNorm | 1,940 linked active drugs; source_data present on nearly all rows but last updated 2024-11-18 | RxNormService and RxNormMatching::Task still write SourceDrug.rxnorm rows (../bioloupe-data-gov/app/services/rx_norm_service.rb, ../bioloupe-data-gov/app/tasks/rx_norm_matching/task.rb) | candidate_live public reference vocabulary; the implemented path enriches an existing identity rather than minting from a lookup response. |
| ChEMBL | 3,601 linked active drugs, but source rows are old/static and source_data is empty | no current source-drug importer; README says ChEMBL DB is read-only reference and not source of truth (../bioloupe-data-gov/README.md) | mapping_only_remainder; expose only as deprecated/mapping external ids, never as identity mint source. |
| Adis | 6,739 linked active drugs, all source rows created/updated 2024-06-20 with no source_data | no active importer found; only FK/admin/search use and a static total_drugs_adis.json artifact | mapping_only_remainder until a current licensed source contract exists; do not treat Adis as an authoritative minting source. |
The implemented mint policy is deliberately narrower than legacy source_drugs: an exact regulator active-substance/ingredient claim or reviewed curation can create the base identity. NCIt/RxNorm snapshots enrich that existing identity with official ids and aliases; ambiguous or colliding authority results route to review. ChEMBL and Adis may appear only in mapping/legacy external-id surfaces unless a future source contract deliberately promotes them.
No source_drugs rule. Legacy source_drugs is a collapsed cross-reference/blob table and must not exist in the greenfield model. Its useful meaning is split into typed rows: vocabulary/source identifiers go to canonical.drug_external_ids, names go to canonical.drug_aliases, and source payload snapshots stay in source/migration tables.
No legacy canonical producer rule. Legacy Data Gov public.drugs, public.source_drugs, and public.brand_drugs are not primary sources. Stage them under migration.* only so old ids, aliases, source-code mappings, and review context can be matched against source-derived identities. They must not insert or update canonical.drugs, canonical.drug_aliases, canonical.drug_external_ids, or canonical.drug_brands by themselves. Canonical minting remains source/curation controlled: FDA/EMA/KEGG/CDE active-substance claims or reviewed curation create base identities; NCIt/RxNorm enrich them; Adis/ChEMBL stay mapping-only unless a current source contract promotes them.
Legacy Data Gov Drug Source Map
Section titled “Legacy Data Gov Drug Source Map”Legacy Data Gov did not get its canonical drugs from FDA approvals. It had one old bulk construction path, then several later maintenance/linkage paths. The old bulk path lived mostly in ../bioloupe-ai-solutions, not in the Rails workflow:
flowchart TD Adis["Adis scrape/export + legacy source_drugs(source=adis)\npaid/proprietary names, synonyms, profiles"] Chembl["ChEMBL local DB / synonym dump\nmolecule ids, preferred names, synonyms, phase"] MatchA["match_chembl_to_adis_by_name\nAdis names/synonyms -> ChEMBL candidates"] MatchB["match_adis_to_chembl\nbest ChEMBL molecule per Adis drug"] Ncit["NCIt EVS API search\ncodes, names, synonyms"] Bundle["chembl_adis_to_ncit.json\nlegacy matched identity bundle"] Import["import_drugs / data-lake import-drugs\nupsert source_drugs + insert/update drugs"] SourceDrugs["source_drugs\nsource=adis|chembl|ncit"] Drugs["drugs\nBioloupe id + source FK ids + all_synonyms"] Adis --> MatchA Chembl --> MatchA MatchA --> MatchB Adis --> MatchB MatchB --> Bundle Ncit --> Bundle Bundle --> Import Import --> SourceDrugs Import --> Drugs SourceDrugs --> Drugs
Evidence:
../bioloupe-ai-solutions/data-collection/build-drug-list.shrunsmatch_chembl_to_adis_by_name,match_adis_to_chembl,match_chembl_adis_to_ncit, thenimport_drugs.../bioloupe-ai-solutions/data-collection/src/drugs_matches/match_chembl_to_adis_by_name.jsreads Data Govsource_drugs where source='adis', combines Adis names/synonyms with ChEMBL synonym data, and assigns legacyBids.../bioloupe-ai-solutions/data-collection/src/drugs_matches/match_adis_to_chembl.jsqueries ChEMBLmolecule_dictionary/molecule_synonymsby Adis names and prefers exact preferred-name and higher-phase matches.../bioloupe-ai-solutions/data-collection/src/drugs_matches/match_chembl_adis_to_ncit.jscalls the NCIt EVS search API for the combined synonym set and writeschembl_adis_to_ncit.json.../bioloupe-ai-solutions/data-collection/src/drugs_matches/import_drugs.jsand../bioloupe-ai-solutions/data-lake/import-drugs/index.jsupsertsource_drugsforadis,chembl, andncit, then insert/updatedrugs.- Legacy artifacts include a 5.4k-line
drugs_list_from_adis.csv/list_of_unique_drugs.csv; the 118k-line clinical-trial drug-name list was used for exploration/enrichment, not as the final Railsdrugsminting path.
Rails-era paths after that bulk construction are mostly linkage and maintenance:
| Legacy path | Reads | Writes | Mints drugs? | Greenfield interpretation |
|---|---|---|---|---|
| Old Adis + ChEMBL + NCIt bulk import | Adis scrape/export or source_drugs(source='adis'), ChEMBL local DB/synonyms, NCIt EVS search | source_drugs(adis/chembl/ncit), drugs | yes | Historical construction evidence only. Do not port mechanically; model current source artifacts instead. |
| NCIt concept conversion/admin linking | ncit_concepts / NCIt payloads | source_drugs(ncit), drugs.ncit_id, sometimes a new drugs row | yes in legacy | Useful authority-linking precedent. In the implemented Dagster path this becomes NCIt snapshot -> verified authority claim -> external id/alias enrichment or review on an existing source-derived drug. |
| RxNorm matching | existing drugs, RxNorm/RxNav service results | source_drugs(rxnorm), drugs.rx_norm_id, RxNorm hierarchy rows | mostly no; links existing drugs | Public vocabulary enrichment/linkage after an identity exists, unless a future explicit RxNorm source policy allows minting. |
| Salt / parent-child cleanup | existing source_drugs hierarchies | drugs.parent_id, occasional parent/child drug rows | cleanup-derived | Resolution/curation operation, not bootstrap. It can propose split/parent links with verifier evidence. |
| Regulatory matching: FDA/EMA/KEGG/CDE/Purple Book | regulatory product/application records | source record drug_id, drug_approvals.drug_id, brands/approval facts | no | Approval/product evidence. It resolves to the drug spine; it does not define the spine. |
| Brand and label workflows | drug_approvals, labels, active substances, brand names | source-grounded active-substance and brand claims | yes for exact active-substance identity; no for a brand name alone | Brand/product layer owned by this spine plus the approvals slice. |
| ActiveAdmin/manual edits | human-entered Drug / SourceDrug records | drugs, source_drugs, locks | yes, manually | Reviewed curation overlay with provenance; not hidden extraction logic. |
Therefore the greenfield source graph treats the legacy catalog as reconciliation evidence, not as a bootstrap catalog. FDA approvals, labels, Purple Book, EMA, KEGG, and CDE provide active-ingredient/product/brand claims that create or resolve the base spine. NCIt/RxNorm snapshots then add authority ids and aliases. Legacy rows preserve continuity only after matching a source-derived canonical identity or becoming reviewed curation input.
Legacy Brand Drug Findings
Section titled “Legacy Brand Drug Findings”Legacy brand_drugs is a derived rollup/cache over regulatory approvals, not a source-authoritative brand catalog. Data Gov builds approval rows first, then groups them by drug_approvals.brand_name, canonicalizes through existing BrandDrug.find_canonical, and stores one selected approval per region (usa_approval_id, eu_approval_id, jp_approval_id, cn_approval_id) (../bioloupe-data-gov/app/workflows/drugs_workflow.rb, ../bioloupe-data-gov/lib/tasks/regulatory/brands.thor, ../bioloupe-data-gov/app/models/drug_approval.rb). Regulatory brand names enter through DrugApproval mappers:
| Approval source | Region | Brand field used by Data Gov |
|---|---|---|
FdaDatum / openFDA | USA | fda_datum.brand_name, filtered to product records with the same brand name. |
EmaApprovedDrug | EU | medicine_name. |
KeggApprovedDrug | JP | brand_name. |
ChinaApprovedDrug / CDE | CN | normalized product_name via Regulatory::CdeBrandNameNormalizer. |
PurplebookRecord | USA | proprietary_name, falling back to proper_name when proprietary name is blank or placeholder. |
The legacy model is intentionally not copied because it collapses four grains into one mutable row:
- brand/trade-name identity (
name,aliases); - marketed/regulatory product facts (
active_ingredient,dosage_form,strength,route,company,atc_code,inn_key); - representative regional approvals (one best approval per USA/EU/JP/CN, selected by approval status/originality/label presence);
- operational search/review/sales anchor (
news_drug_mentions,brand_sales, ActiveAdmin review surfaces).
Read-only legacy sampling on 2026-06-30 confirms this is dirty enough to require salvage semantics: active brand_drugs = 13,118; linked to a legacy drug = 3,663; rows with zero regional approval links = 3,897; rows with aliases = 1; duplicate normalized brand-name groups = 303 covering 726 rows; approval brand names linked to multiple legacy drugs = 14; rows where a selected regional approval points at a different drug than brand_drugs.drug_id = 8. Active approval brand-name source counts were FdaDatum USA 27,322, EmaApprovedDrug EU 2,326, ChinaApprovedDrug CN 2,046, KeggApprovedDrug JP 900, and PurplebookRecord USA 687.
Greenfield treatment:
migration.legacy_drug_brand_baselinestagespublic.brand_drugsas a provenance-bearing salvage baseline, including all four regional approval links as structured context, not as the canonical model shape.canonical.drug_brandsis only brand/trade-name identity linked to a canonical drug. It must not copy approval/product attributes as brand identity.- Approval/product facts stay with the regulatory approvals/product slice and can expose many-to-many brand-to-approval relationships in query models. Do not model one fixed approval column per region.
- Unmatched legacy brand rows remain staged for review or future source-derived backfill. They do not mint canonical drugs or brand links by themselves.
Full Materialization Plan
Section titled “Full Materialization Plan”The complete implementation is source-first. Legacy rows are a reconciliation side input, never the canonical producer:
flowchart TD Ncit["NCIt source snapshot\nsource vocabulary"] RxNorm["RxNorm/RxNav source snapshot\nsource vocabulary"] Reg["FDA/EMA/KEGG/CDE approval snapshots\nproduct/brand/ingredient claims"] Mentions["trial/publication/news/SOC mentions\nidentity demand"] Legacy[(Legacy Data Gov RDS\nread-only)] Bridge["migration.legacy_drug_baseline\nold ids + stale mappings"] Claims["evidence.drug_identity_claims\nalias/id/brand/product claims"] Resolve["resolution.drug_intervention_candidates\n+ verifier/review"] CanonDrug["canonical.drugs"] CanonAlias["canonical.drug_aliases"] CanonIds["canonical.drug_external_ids"] CanonBrand["canonical.drug_brands"] CanonCombo["canonical.drug_combinations\n+ components"] Names["resolution.entity_names\nentity_type=drug"] Query["q_drugs family"] Ncit --> Claims RxNorm --> Claims Reg --> Claims Mentions --> Resolve Legacy --> Bridge -. reconciliation only .-> Resolve Claims --> Resolve Resolve --> CanonDrug Resolve --> CanonAlias Resolve --> CanonIds Resolve --> CanonBrand Resolve --> CanonCombo CanonDrug --> Names CanonAlias --> Names CanonBrand --> Names CanonDrug --> Query CanonAlias --> Query CanonIds --> Query CanonBrand --> Query CanonCombo --> Query
Source / Migration Assets
Section titled “Source / Migration Assets”Source-derived implemented assets:
| Asset | Reads | Writes | Purpose |
|---|---|---|---|
evidence/drug_identity_claims | normalized.approval_source_records | evidence.drug_identity_claims | Extract primary-source active-substance, source-alias, and brand-name claims from regulatory approval/product records. Product brand claims link only when the source row has exactly one active substance. |
sources/drug_authority/lookup_snapshots | evidence.drug_identity_claims substance terms | sources_drug_authority.lookup_snapshots, provenance.source_artifacts | Cache-first official NCIt/RxNorm captures for observed drug terms. The repository selects missing (source_family, normalized_term) requests before any live max_live_records bound; the authority_source resource owns fixture/live mode and HTTP adapters. Raw authority payloads stay in source tables; this is not a recreated source_drugs table. |
evidence/drug_authority_claims | sources_drug_authority.lookup_snapshots | evidence.drug_identity_claims | Parse exact authority matches into external_id claims and source aliases. No fuzzy or closest-result promotion. |
canonical/source_drug_identity_claims | approval/product evidence.drug_identity_claims | canonical.drugs, canonical.drug_aliases, canonical.drug_brands | Promote source-derived approval/product claims into the base canonical drug and brand identity surface. Legacy rows are not read by this asset, and NCIt/RxNorm lookup snapshots are not a prerequisite for approval-product resolution. |
canonical/drug_authority_external_ids | evidence.drug_authority_claims, canonical/source_drug_identity_claims | canonical.drug_external_ids, canonical.drug_aliases, resolution.review_queue | Enrich existing same-normalized source-derived drugs with official NCIt/RxNorm external ids and source aliases; external-id collisions route to review. It does not mint approval-local identity, copy legacy source_drugs, or block canonical.drug_brands / resolution.regulatory_entities. |
Legacy reconciliation assets:
| Asset | Reads legacy | Writes local | Purpose |
|---|---|---|---|
sources/legacy_data_gov/drug_baseline | public.drugs d plus left joins to public.source_drugs for d.ncit_id, d.rx_norm_id, d.adis_id, d.chembl_id | migration.legacy_drug_baseline | Reconcile old Bioloupe ids, names, and stale source-code mappings to source-derived canonical drugs; never mint canonical rows. |
sources/legacy_data_gov/drug_brand_baseline | public.brand_drugs plus linked drug_approvals context | migration.legacy_drug_brand_baseline | Preserve legacy brand review/search context while treating region approvals and product facts as source context, not canonical brand shape. |
sources/legacy_data_gov/drug_combination_baseline | public.combination_drug_components, public.drugs.parent_id, drugs.is_combination | migration.legacy_drug_combination_baseline | Reconcile legacy combination/regimen shells to source-derived component drugs; unmatched rows stay staged. |
sources/legacy_data_gov/drug_target_baseline | public.drug_target_actions joined to legacy drugs/targets | migration.legacy_drug_target_baseline | Transitional reconciliation input until the target/technology spine fully owns this. |
All four assets use the disabled-by-default legacy_data_gov_source, are bounded by its
max_import_records config, enforce read-only database sessions, and keep source_locator +
content_hash. Their output is useful for audit, continuity matching, and coverage reports only.
The drug baseline query must filter out soft-deleted rows from publication surfaces by default
while still allowing an audit mode to stage deleted_at rows for reconciliation reporting.
Topology Decision: Base Identity vs Authority Enrichment
Section titled “Topology Decision: Base Identity vs Authority Enrichment”Approval-derived active-substance and brand claims are the base identity surface: they create the canonical drug/brand rows and the resolution.entity_names entries that approvals, trials, publications, SOC, and news can resolve against. NCIt/RxNorm are authority enrichment sources for external ids after a source-derived drug exists. The source asset is intentionally thin: it asks AuthoritySourceRepository for pending missing requests, asks authority_source for fixture/live capture, then persists immutable snapshots and source-artifact receipts. This keeps the Dagster graph faithful to the domain: FDA/EMA/KEGG/CDE product records can resolve their own product_name/generic_name without waiting for live NCIt/RxNorm lookups, while q_drugs.primary_external_ids still remains source-backed by official vocabulary snapshots.
Rejected alternative: one monolithic canonical/source_drug_identity_claims asset that waits on NCIt/RxNorm before rebuilding brand/name resolution. That couples a useful enrichment path to approval publication and makes an authority lookup a hidden precondition for source-product resolution. Legacy Data Gov did the equivalent coupling through mutable source_drugs FKs and brand_drugs rollups (../bioloupe-data-gov/app/models/source_drug.rb, ../bioloupe-data-gov/app/models/brand_drug.rb, ../bioloupe-data-gov/app/models/drug_approval.rb); greenfield keeps those grains separate.
migration.legacy_drug_baseline schema:
| Column | Type | Meaning |
|---|---|---|
legacy_drug_id | bigint primary key | public.drugs.id. |
bioloupe_id | text | Legacy stable B... id, nullable. |
drug_name | text not null | public.drugs.name; rows with blank names are rejected to ops.rejected_records. |
inn_name | text | public.drugs.inn_key. |
all_synonyms | jsonb not null default '[]' | Raw legacy synonym array for reconciliation/review; not direct alias authority. |
legacy_status | text | Legacy approval/workflow status (status). |
legacy_qa_status | text | Legacy QA status. |
drug_status | text | Legacy lifecycle summary retained for reconciliation only; not canonical development truth. |
development_phase | text | Legacy development summary retained for reconciliation and program-gap analysis only; not an identity key or canonical phase. |
drug_type | text | Legacy regulatory/product-class facet used by ownership-suggestion filters; staged separately and not forced into modality. |
therapeutic_areas | jsonb not null default '[]' | Legacy therapeutic-area array used by ownership-suggestion and disease/indication handoff; not a canonical drug-identity field. |
is_combination | boolean not null default false | Legacy is_combination. |
deleted_at | timestamptz | Soft-delete marker; normally excluded from reconciliation dashboards. |
external_ids | jsonb not null default '{}' | Source-code map recovered from left-joined source_drugs: {ncit, rxnorm, adis, chembl}. Values are source codes, not legacy FK ids. |
source_aliases | jsonb not null default '{}' | Optional source-row names/synonyms recovered from joined source_drugs, used only as candidate alias evidence. |
source_locator | jsonb not null | {source_family:'legacy_data_gov', legacy_table:'public.drugs', legacy_pk, bioloupe_id} plus joined locator metadata. |
content_hash | text not null | Hash over the staged legacy fields used for idempotency/drift. |
captured_at | timestamptz not null default now() | Capture time. |
The source asset must reject only row-local failures to ops.rejected_records (source_family='legacy_data_gov', stage='legacy_drug_baseline'). Required source-stage reason codes: legacy_drug_blank_name, legacy_drug_bad_synonyms. Cross-row identity conflicts belong to source-derived resolution/review, where normalized-name and external-id ownership maps exist. Do not create a one-off reject table.
Canonical Assets
Section titled “Canonical Assets”| Asset | Reads | Writes | Rule |
|---|---|---|---|
canonical/source_drug_identity_claims | Source-derived drug identity claims from approvals/products | canonical.drugs, canonical.drug_aliases, canonical.drug_brands | Mint/link only from approved source policies and verifier/curation receipts; does not depend on NCIt/RxNorm enrichment. |
canonical/drug_authority_external_ids | Official NCIt/RxNorm external-id and alias claims for already source-derived drugs | canonical.drug_external_ids, canonical.drug_aliases, resolution.review_queue | Attach source-contract-backed external ids and aliases after the base identity exists; collisions route to review rather than overwriting. |
canonical/source_drug_combinations | combination/regimen claims, component drug identities | canonical.drug_combinations, canonical.drug_combination_components | Combination shells reference component drugs; never collapse components into one substance. |
canonical/drug_identity_names | canonical drug/alias/brand tables | resolution.entity_names | Rebuild the resolver surface after source-derived canonical rows materialize. |
The target/technology spine owns target authority capture, target/mechanism claims, and
canonical.drug_target_action; drug identity contributes only the resolved drug endpoint.
Canonical row ids and keys:
drug_idis deterministic from the source-authority identity claim, e.g. source family + source id + release-grain policy, so reruns are idempotent without using legacy primary keys.resolved_drug_keyis an opaquedrug:<slug-or-bioloupe-id>surface. A legacyB...id may be retained only after reconciliation links it to a source-derived identity.source_fact_keypoints to the source artifact/evidence claim that justified the canonical assertion, not topublic.drugs.confidenceis a categorical provenance basis such assource_cross_reference,verifier_passed, orreviewed_curation, not a numeric score.
Canonical write policy:
- upsert
canonical.drugsby deterministic source-deriveddrug_id; - never update a canonical row from a legacy migration table;
- insert aliases with
ON CONFLICT DO NOTHINGafter deterministic normalization; - insert external ids with
ON CONFLICT (source, external_id) DO NOTHING; if the same source id already maps to a different drug, route the claim to review instead of overwriting; - treat duplicate normalized primary names as review unless the duplicate has the same vetted source id map or an approved merge curation decision.
Legacy Reconciliation Mapping
Section titled “Legacy Reconciliation Mapping”| Legacy input | Greenfield use |
|---|---|
drugs.id | retained only in source_locator.legacy_pk and reconciliation reports. |
drugs.bioloupe_id | old stable product key to attach to a source-derived resolved_drug_key after match/review. |
drugs.name | candidate display-name evidence for matching; never the sole canonical primary-name authority. |
drugs.inn_key | candidate alias/INN evidence after it matches a source-derived drug. |
drugs.all_synonyms | candidate alias evidence after source-derived match or reviewed curation; no direct alias insert. |
drugs.drug_status, drugs.status | legacy workflow context for review queues and continuity dashboards; no direct assertion-state policy. |
drugs.development_phase, drugs.drug_type | preserved in migration.legacy_drug_baseline for reconciliation and coverage analysis only; do not overload canonical.drugs.modality or use as ownership/development truth. |
drugs.therapeutic_areas | preserved in migration.legacy_drug_baseline for organisation-ownership candidate generation and later disease/indication reconciliation; not written as canonical drug identity. |
drugs.is_combination, combination_drug_components, parent_id | candidate combination evidence; canonical combinations require source-derived component drugs or reviewed curation. |
joined NCIt/RxNorm source rows code | reconciliation key to current NCIt/RxNorm snapshots; canonical external id only when the current source artifact confirms it. |
joined Adis/ChEMBL source rows code | mapping-only/deprecated traceability after a source-derived drug exists; never a minting source. |
joined source row name / synonyms | candidate alias evidence after matching; never a source_drugs table. |
brand_drugs.name, approval brand_name | brand candidate evidence; canonical brand rows come from approval/product source records and must resolve to a source-derived canonical drug. |
Assertion-State Policy
Section titled “Assertion-State Policy”| Source-derived state | canonical.drugs.assertion_state | Use |
|---|---|---|
| active source identity later withdrawn/deprecated by source or curation | deprecated | Hidden from publication q_drugs unless explicitly requested. |
| vetted NCIt/RxNorm/source-artifact identity with verifier/review support | resolved | Published identity. |
| source-supported but unresolved/conflicting identity | provisional | Searchable for review, but cannot auto-merge or receive legacy aliases without evidence. |
Minting Policy
Section titled “Minting Policy”Sources never write directly into canonical.drugs by pretending to be legacy rows:
- NCIt/RxNorm source snapshots create vocabulary records and identity/alias/external-id claims.
- Resolver links claims to existing source-derived identities by exact source id/name/alias.
- Resolver mints only from an exact accepted active-substance claim or reviewed curation when no existing normalized identity matches.
- Curation can merge/split/repoint identities through reviewed append-only decisions.
- Adis/ChEMBL ids imported from legacy remain mapping-only external ids.
Ownership-Suggestion Handoff
Section titled “Ownership-Suggestion Handoff”Latest legacy Data Gov runs drug ownership suggestions immediately after therapeutic-area refresh (../bioloupe-data-gov/app/workflows/drugs_workflow.rb) and filters ownerless drugs by drug_type, drug_status, development_phase, therapeutic_areas, plus approval/trial evidence (../bioloupe-data-gov/app/services/ownership_suggestions/drug_sync_service.rb). Greenfield preserves those legacy fields in migration.legacy_drug_baseline only to explain legacy coverage and compare candidate populations. The organisation slice rebuilds ownership candidates from explicit source roles and temporal relationship evidence; legacy lifecycle summaries are not an acceptance basis.
Drug identity still does not own ownership suggestions. 26-slice-organisation-intelligence.md owns temporal asset ownership (canonical.asset_ownerships, q_asset_ownerships) and the reviewed/verified current-owner decision. This spine only supplies the resolved drug key and source-backed drug context.
Checks and Tests
Section titled “Checks and Tests”Blocking checks:
no_source_drugs_table— no greenfield schema creates asource_drugstable.no_legacy_drug_canonical_producer— no Dagster asset writesmigration.legacy_drug_baselineinto canonical drug tables.source_drug_identity_claims_present— source approval/product records produceevidence.drug_identity_claimsbefore canonical promotion.legacy_drug_reconciliation_coverage— staged legacy rows report matched/unmatched/review counts against source-derived canonical identities.source_external_id_coverage— NCIt/RxNorm ids incanonical.drug_external_idspoint to current source snapshots, not just legacy joined codes.identity_uniqueness— normalized names remain unique or the claim is routed to review.legacy_reject_dlq— all skipped baseline rows are present inops.rejected_records, not silently dropped.no_separator_component_auto_link— intervention resolution never splits/,+, comma, semicolon, orandand publishes one arbitrary component as the whole intervention.
Minimal implementation tests:
- unit test that staged legacy drug rows have no canonical-producing asset;
- unit test that regulatory active-substance rows create source-derived substance, alias, and brand claims;
- unit test that multi-substance product rows do not brand-link to one arbitrary component;
- unit test that NCIt/RxNorm claims enrich existing identities and route missing/colliding identities to review;
- unit test that a source-supported but unresolved active drug becomes
provisional; - unit test that ChEMBL/Adis external ids are deprecated/mapping-only;
- unit test that an external-id collision routes to review and does not overwrite the existing canonical mapping;
- unit test that
"drug A + drug B"resolves only to an existing combination/regimen row or remains unresolved/review, never to just one component; - definition/SQL test that no
source_drugsrelation is created; - DB smoke test materializing source-derived drug identity ->
q_drugs, with legacy reconciliation remaining sidecar-only.
Slice-Local Domain Model
Section titled “Slice-Local Domain Model”Distinct grains (do not collapse):
| Concept | Grain | Legacy evidence | Notes |
|---|---|---|---|
| Active substance / drug | one compound at INN level | drugs (name, inn_key, all_synonyms) | canonical identity; combination_drug_components self-join for fixed-dose combinations. |
| Drug alias | one name variant for a drug | drugs.all_synonyms, brand_drugs.aliases | every variant must trace to one drug. |
| External id | one source-scoped identifier | drugs.ncit_id/rx_norm_id/adis_id, plus legacy chembl_id when salvaged | Source locators, not global identity. NCIt/RxNorm are vetted public reference ids; Adis/ChEMBL are mapping-only legacy/proprietary ids unless future source contracts promote them. |
| Branded product | one marketed brand/trade name linked to a canonical drug | brand_drugs.name, brand_drugs.aliases, and linked approval brand_name values | legacy brand_drugs is a rollup/cache; product attributes and selected regional approvals stay with approval/product facts. |
| Combination / regimen | a set of component drugs as a treatment unit | combination_drug_components, chemo_combination | a combination is not a new molecule. |
| Intervention component | one arm/protocol intervention with dose/route/schedule | trial_arm_interventions, publication_interventions | resolves to drug, NCIt concept, combination, or unresolved. |
| Non-drug intervention | procedure/device/radiation/behavioral/placebo | NCIt concepts via ncit_concept_id | must not resolve to a drug (issue 50). |
| Target / mechanism / modality | what a drug acts on / how it is built | drug_target_action, target, technology | identity and hierarchy are owned by 31-slice-target-technology-identity.md; this spine owns only the drug endpoint of the relation. |
Invariants
Section titled “Invariants”- one canonical drug per compound; alias/external-id/brand variants point to it, never duplicate it;
combination/regimenreference component drugs and never replace them;- an intervention component resolves to exactly one winning identity (drug xor NCIt concept) or stays unresolved — mirrors the legacy single-FK rule (
publication_interventions.drug_idorncit_concept_id, never both;../bioloupe-data-gov/CLAUDE.md); - external ids (
ncit/rx_norm, plus mapping-onlyadis/deprecatedchemblmappings when salvaged) are source-scoped and not interchangeable; - generic class terms (“statins”, “PD-1 inhibitor”, “CAR-T”) must not auto-resolve to a specific product (
drug.rbMATCH_STOPLIST/generic_class_term?).
Slice-Local Entity Resolution
Section titled “Slice-Local Entity Resolution”This spine owns the canonical resolver that trials/approvals/publications/news/SOC consume. It is exact-first, then LLM-verified fuzzy, with discrete provenance basis categories — never a bare trigram auto-accept (../bioloupe-data-gov/app/services/entity_matcher.rb, ../bioloupe-data-gov/CLAUDE.md “never bare flexifind”).
| Subject kind | Source term examples | Candidate target types | Publishable basis | High-impact? |
|---|---|---|---|---|
intervention_drug_link | pembrolizumab, KEYTRUDA, MK-3475, imatinib mesylate | canonical Drug (incl. broader salt/formulation match) | exact synonym, source cross-reference, context-verified, verifier-passed fuzzy/LLM, reviewed curation override | yes |
intervention_concept_link | surgery, EBRT, PET/CT, placebo, exercise | NCIt non-drug concept | NCIt exact/semantic-scoped, verifier-passed LLM, reviewed override | yes |
brand_product_link | Keytruda, Tagrisso | canonical BrandDrug -> Drug | normalized brand/alias exact, source cross-reference, reviewed override | yes |
drug_alias_promotion | term confirmed = drug across many sponsors | new alias on canonical Drug | offline reviewed promotion only (entity_matching promote_synonyms) | yes |
combination_membership | ”pembrolizumab + chemo” | combination -> component drugs | source-exact component listing or verifier/reviewed override | medium/high |
Resolver guards that must survive into greenfield (all from drug_matching_service.rb):
- formulation guard — liposomal/pegylated/etc. must not collapse onto the plain molecule;
- radioisotope guard — a different isotope is a different agent (Y-90 ≠ Ga-68);
- sponsor/context disambiguation — owner organisation breaks ties only when the name cannot identify the product;
- broader match — salt/dose/formulation variant may match the base drug as
match_type=broader; - whole-intervention guard — the resolver resolves the full intervention string as one term and does not split on separators to choose a single component. Latest legacy Data Gov removed that fallback in
../bioloupe-data-gov/app/services/intervention_resolver.rbbecause it corrupted dosing units and collapsed fixed-dose combinations to arbitrary constituents. A combination/regimen either resolves to a combination/regimen identity or stays unresolved for review.
Evidence Claim Schemas
Section titled “Evidence Claim Schemas”drug_identity_claim
Section titled “drug_identity_claim”A source-grounded assertion that a name denotes a specific substance/product, with source locator and extractor/model version. Promoted to canonical identity only after resolution/verifier acceptance.
drug_alias_claim
Section titled “drug_alias_claim”A name-variant-for-drug claim (synonym/brand/code) with source authority; feeds alias resolution, not silent merge.
drug_external_id_claim
Section titled “drug_external_id_claim”A source-scoped id mapping (ncit/rx_norm, plus mapping-only salvaged ids such as adis and deprecated chembl) with the source release/version and liveness state it came from.
intervention_resolution_claim
Section titled “intervention_resolution_claim”A claim that a trial/publication intervention string resolves to a drug or NCIt concept, with basis category and verifier state.
Curation Override Subjects
Section titled “Curation Override Subjects”Reviewed overrides correct identity after pipeline inference; they are not the default path (separate from LLM extraction per 05-agentic-curation-and-overrides.md).
| Subject | Why high-impact | Example |
|---|---|---|
drug_merge_override | wrong split fragments a compound’s evidence | merge two drugs rows for the same compound. |
drug_split_override | wrong merge contaminates two compounds | split an over-merged drug. |
intervention_identity_override | wrong link poisons every downstream program view | force “MK-3475” -> pembrolizumab, or reject a non-drug from a drug match. |
brand_to_drug_override | wrong brand link corrupts approvals/competitive landscape | re-point a brand to the correct drug. |
alias_promotion_override | bad alias becomes a permanent false match | approve/revoke a synonym promotion. |
Legacy preserves manual curation against automated overwrite via Lockable/lockable_attributes and durable manual matches (entity_match_caches.provenance='manual'); greenfield expresses this as reviewed overlays, not Rails locks.
Canonical Assets
Section titled “Canonical Assets”Durable nouns (group canonical, per 04-dagster-asset-graph.md):
canonical.drugs— one compound (INN-level) with identity assertion state; no global development phase/status.canonical.drug_aliases— one alias per drug.canonical.drug_external_ids— source-scoped id mappings.canonical.drug_brands— branded/marketed products linked to a drug.canonical.drug_combinations/canonical.drug_combination_components— combination/regimen membership.resolution.drug_intervention_candidates— proposed intervention->identity links with basis + verifier state.
Target/mechanism/technology identity is owned by 31-slice-target-technology-identity.md; this spine publishes resolved drug keys used by those relations.
q_drugs Family
Section titled “q_drugs Family”All read-only SQL models in the query schema; visible in generated q_catalog, provenance-required rows resolve through q_provenance.
| Model | Grain | Agent question |
|---|---|---|
q_drugs | one canonical drug | drug search by name/synonym/INN; identity assertion state, modality summary, and key external ids. Contextual development state is joined through q_development_programs. |
q_drug_aliases | one alias | name/synonym lookup and identity trace. |
q_drug_external_ids | one source-scoped id | cross-reference to NCIt/RxNorm plus mapping-only Adis/ChEMBL traceability; only source-contract-backed ids may be promoted into q_drugs.primary_external_ids. |
q_drug_brands | one branded product with source approval/product context | brand -> drug resolution, marketed-name search, and compact FDA/EMA/KEGG/CDE source-product backing. |
q_drug_regimens | one combination/regimen + components | what a combination contains. |
Drug-target relations are owned and published by the target/technology spine as
q_drug_target_actions; the drug identity family does not expose a second transitional copy.
resolved_drug_key(s) used across 21-35 is defined here as a stable key into q_drugs (e.g. drug:<slug>). Downstream slices reference this key; they do not mint their own drug identity.
q_drugs contract seed
Section titled “q_drugs contract seed”| Field | Required content |
|---|---|
| Purpose | Canonical drug discovery and the resolution target for every drug mention. |
| Grain | One canonical compound. |
| Stable row key | resolved_drug_key (drug:<slug>), used by q_provenance.q_row_key. |
| Required columns | resolved_drug_key, drug_name, inn_name, synonyms (text[]), modality, is_combination, primary_external_ids (jsonb: vetted live ids, currently ncit/rxnorm), legacy_external_ids (jsonb: mapping-only ids such as adis/chembl), assertion_state, provenance_key. |
| Search affordances | trigram/full-text on drug_name/synonyms; GIN on synonyms; btree on assertion_state/modality. |
| Provenance rule | provenance_grain=bundle; identity bundle expands to source vocabulary artifact, resolution/verifier receipt, canonical assertion, and curation overlay only when an override exists. |
| Refresh | materialized view/table; full refresh, targeted refresh on reviewed identity overrides. |
| Blocking checks | schema_contract, provenance_coverage, identity-uniqueness (one canonical row per compound). |
Checks and Quality Gates
Section titled “Checks and Quality Gates”- every
resolved_drug_keyin anyq_*model exists inq_drugs(enforced corpus-wide by thereferential_integritygate in08-contracts-and-quality.md); - no two canonical drugs share a normalized identity (uniqueness like
drug.rbnormalize_name); - an intervention component never resolves to both a drug and a non-drug concept;
- generic class terms do not resolve to specific products;
- no
q_drugsfield is documented or consumed as a global clinical phase, clinical activity, regulatory, or “highest phase ever” state; q_drugsfamily schema matches generatedq_catalog; provenance-required rows resolve throughq_provenance.
Concepts Promoted to 20-pipeline-families.md
Section titled “Concepts Promoted to 20-pipeline-families.md”- Drug identity is a foundational spine consumed by all therapeutic slices; the cross-slice “Drug / intervention” concept resolves to this spine’s
resolved_drug_key. - Substance vs brand vs combination vs intervention vs non-drug concept are distinct grains.
Locked Contract
Section titled “Locked Contract”resolved_drug_keyis the deterministicdrug:<normalized-slug>key for the current contract; changing it requires an explicit key migration and compatibility plan.- Combination/regimen identity is first-class through
q_drug_regimens; downstream slices must not encode regimen membership in free text. - Clinical phase, activity, regulatory state, and historical maximum are contextual development-program facts owned by
33-slice-development-landscapes.md, never drug identity columns.