Drug approvals pipeline

When the FDA approves a new cancer drug, that approval appears in Data Gov within days — matched to the correct Bioloupe drug, linked to structured indications, and enriched with label text. This page follows a drug approval from raw regulatory data through the pipeline to a unified DrugApproval record.

Four agencies, one unified model

The same drug receives separate approvals in each region. Pembrolizumab is approved by the FDA (USA), EMA (EU), KEGG/PMDA (Japan), and CDE/NMPA (China). Each agency publishes data in a different format, with different identifiers, at different times.

Data Gov collects from all four and assembles a single drug_approvals record per region per drug. Polymorphic source_type/source_id columns trace every approval back to its raw regulatory source.

flowchart TB
  subgraph Sources["Four Regulatory Agencies"]
    FDA["FDA\n(openFDA API)\nNDA/BLA numbers"]
    EMA["EMA\n(XLSX export)\nProduct numbers"]
    KEGG["KEGG\n(Node.js scraper)\nKEGG identifiers"]
    CDE["CDE/NMPA\n(Database)\nCDE codes"]
  end

  subgraph Match["Match to Bioloupe Drugs"]
    MFDA["Match FDA records"]
    MEMA["Match EMA records"]
    MKEGG["Match KEGG records"]
    MCDE["Match CDE records"]
  end

  subgraph Assembly["Assembly"]
    ASM["Create unified\nDrugApproval records"]
    PB["Purple Book\nbiosimilar approvals"]
    DIT["Determine\ninnovation type"]
    LNK["Link organisations\nto approvals"]
  end

  subgraph Enrich["Enrichment"]
    LBL["Fetch FDA labels\n(DailyMed)"]
    BW["Parse boxed\nwarnings"]
    REMS["Fetch REMS\nsafety data"]
    CDX["Sync companion\ndiagnostics"]
    STR["Structure labels\n(LLM)"]
  end

  subgraph Indications["Indication Extraction"]
    PI["Parse indications\n(LLM)"]
    PPD["Link diseases"]
    PPB["Link biomarkers"]
    ETA["Extract therapeutic\nareas"]
  end

  FDA --> MFDA --> ASM
  EMA --> MEMA --> ASM
  KEGG --> MKEGG --> ASM
  CDE --> MCDE --> ASM
  ASM --> PB --> DIT --> LNK
  LNK --> LBL --> BW --> REMS
  LBL --> STR --> PI --> PPD & PPB
  STR --> ETA
  LNK --> CDX

Data sources in detail

FDA (USA)

Source: openFDA download API (api.fda.gov/download.json)

The Regulatory::Fda Thor task downloads partitioned ZIP files containing JSON records for drugs and labels. Each record represents an FDA application (NDA or BLA). The pipeline:

1. Downloads drug partition files. Extracts records matching oncology/hematology therapeutic areas.
2. Downloads label partition files. Matches labels to drugs by application number.
3. Upserts FdaDatum records keyed on application_number + brand_name.

FDA data also feeds supplementary tables:

• Orange Book (orangebook_records) — Patent expiry dates and exclusivity periods
• Purple Book (purplebook_records) — Biological products, biosimilar reference products
• Breakthrough Therapy (breakthroughs) — FDA Breakthrough Therapy designations
• Companion Diagnostics (fda_approved_companion_diagnostics) — Approved diagnostic tests paired with drugs and biomarkers
• REMS (rems) — Risk Evaluation and Mitigation Strategies

Domain context

NDA (New Drug Application) covers small molecules. BLA (Biologics License Application) covers biological products. sNDA/sBLA are supplemental applications for new indications or formulations. These identifier types appear throughout the regulatory data.

EMA (EU)

Source: EMA public XLSX export

The Regulatory::Ema Thor task downloads the spreadsheet and creates EmaApprovedDrug records. Only category: 'Human' records are processed. Fields include authorization status, ATC code, therapeutic area, and regulatory flags (orphan, conditional, accelerated, PRIME, advanced therapy).

KEGG (Japan)

Source: KEGG drug database via Node.js scraper

Creates KeggApprovedDrug records for Japanese approvals. Remark codes indicate regulatory categories: G (generic), B (biosimilar), C (conditional), E/A (expedited review).

CDE (China)

Source: China CDE/NMPA chemical and post-market APIs, plus official CDE label PDFs.

The automated regulatory:cde:collect flow:

1. Syncs chemical and marketed approval records from the CDE APIs.
2. Translates Chinese API fields used by the canonical ChinaApprovedDrug record.
3. Discovers and downloads eligible official CDE label PDFs.
4. Extracts and translates complete label text.
5. Structures label sections and canonical fields, including indications and organisation ownership.

original_data stores the current raw API source snapshot. Extracted Chinese PDF text, its English translation, and processing provenance are stored in dedicated label fields on ChinaApprovedDrug.

DrugsWorkflow runs CDE collection, matching, and approval assembly automatically. regulatory:cde:sync remains available when only the API source records need to be refreshed.

Matching: from source records to Bioloupe drugs

Each source table carries a nullable drug_id FK. The matching step links source records to canonical Drug records using name matching against drugs.name and drugs.all_synonyms.

The matching Thor tasks (regulatory:matching:match_fda_to_bioloupe, etc.) run after collection and before assembly. Unmatched records remain with drug_id: null and queue for human review in ActiveAdmin.

For chemical CDE records, regulatory:matching:match_cde_to_bioloupe matches only on the source-provided English fields (hxcfen, ypmcen, spmen) rather than LLM-translated Chinese names, to avoid translation drift.

Automated matching updates source-record matches as source data and matching logic change. Organisation and partner associations on the assembled DrugApproval can be locked in the admin UI; their dedicated linking tasks preserve those locks. To explicitly reset non-curated source matches before re-running matching, use bundle exec thor one_off:reset_bad_regulatory_matches:execute --regulatory=cde --no-dry_run.

Assembly: creating unified approvals

The regulatory:approvals:assemble task creates unified DrugApproval records from all source tables. For each source record, it:

1. Maps source-specific fields to a common schema (brand name, approval status, region, dates, innovation type)
2. Looks up an existing DrugApproval by source_type + source_id
3. Falls back to matching by application_number + brand_name for manually created records
4. Upserts with lockable attribute support — locked fields are never overwritten
5. Tracks indication changes (old vs. current) in supplementary_info.indications JSONB

Innovation types determine whether an approval is for an original drug, a generic, or a biosimilar. The pipeline derives this from FDA submission class codes, EMA generic/biosimilar flags, and KEGG remark codes.

Innovation type	Meaning
original	First approval of a new molecular entity
generic	Small molecule copy of an approved drug
biosimilar	Biological product similar to an approved biologic
biosimilar_interchangeable	Biosimilar approved as interchangeable with the reference product
non_original	Other non-original products

CDE innovation mapping

CDE records come from two different source shapes:

• Chemical Drug List (category / sllb): this is already a catalog inclusion category. Innovative drugs and Imported original drugs map to original; new-registration generics and consistency-evaluation drugs map to generic; Other medicines stays blank because it mixes older originator and non-originator products.
• Marketed Drug List (drug_type + registerkind / registration_category): registration category is interpreted by drug type. If a record contains multiple numeric categories, mapped values are resolved in this order: biosimilar, generic, non_original, then original.

CDE drug type	Registration categories	innovation_type
Chemical	1, 1.1, 1.2, 2.4, 5.1	original
Chemical	2.* except 2.4, plus 1.3, 1.4, 1.5, 1.6, 2, 5	non_original
Chemical	3.*, 3, 4, 5.2, 6	generic
Therapeutic biological	1	original
Therapeutic biological	imported 2.2 with a JX... application number	original
Therapeutic biological	domestic/non-JX 2.2, 3.1, old 2, old 7	blank
Therapeutic biological	2.* except the 2.2 special case, plus 3.2, 3.4, 4, 5, 6, 8-15	non_original
Therapeutic biological	3.3	biosimilar
Preventive biological	1, 1.*	original
Preventive biological	2, 2.*, 3, 3.2, 3.3, 4, 5, 7-15	non_original
Preventive biological	3.1, 6	blank
TCM	1, 1.*, 3, 3.*, 5, 6, 6.1, 6.2	original
TCM	2, 2.*, 7, 8	non_original
TCM	4, 4.*, 9	generic

Chemical class 5.1 covers overseas-marketed original drugs and clinically advantaged modified drugs, not generics, so it is treated as an originator/innovator approval for this system. Chemical class 2.4 is new indication expansion; local CDE records are overwhelmingly originator or originator-like company follow-on filings, so it is treated as original rather than as a copy. Therapeutic biological imported 2.2 filings map to original because they are new-indication or patient-population follow-ons for imported products such as Keytruda. Domestic therapeutic biological 2.2 stays blank because the category also contains non-originator company filings. The application-number prefix is used only for that therapeutic-biological 2.2 split: JX identifies imported new-drug applications, while CX identifies domestic new-drug applications; it is not used as a standalone innovation classifier. Unknown drug types, missing categories, and categories not listed above stay blank.

Label enrichment

Before assembly, CDE collection downloads, translates, and structures eligible official CDE label PDFs. After assembly, the pipeline also enriches FDA approvals with structured label content.

Fetch FDA labels. Fda::LabelSync calls the DailyMed API to retrieve label metadata and sections. Labels contain the official indication text, dosing, warnings, and adverse reactions.

Parse boxed warnings. Fetches SPL XML from DailyMed and extracts the boxed warning HTML. Boxed warnings are the highest-severity safety alerts the FDA issues.

Fetch REMS. Downloads REMS CSV files from FDA. Creates Rems records with program details, materials, and linked application numbers.

Sync companion diagnostics. Updates fda_approved_companion_diagnostics with paired drug-biomarker-test relationships.

Ownership suggestions for ownerless drugs

After approval organisations are linked and after drug therapeutic areas are updated, the drugs workflow runs clinical_trials:ownership_suggestions:sync --source=approvals.

This step is intentionally drug-centric. It looks for eligible drugs that currently have no current non-partner owner and builds or refreshes an ownership review item in /admin/ownership_suggestions. Approval-linked organisations become one evidence source for the review item; clinical-trial sponsor evidence and Parallel.ai owner suggestions can also be attached when those sources are processed.

This queue does not automatically create ownerships. Reviewers decide whether to:

• assign one or more current owners
• optionally assign regional ownership coverage
• create a missing organisation
• accept a suggested match from an unmatched upstream organisation name to an existing organisation in the system

Default sync filters:

• drug types: New Molecular Entity, New Active Ingredient, New Formulation, Biosimilar, Interchangeable, Original
• therapeutic areas: Oncology, Malignant Hematology, Non-Malignant Hematology
• excluded by status at the ownerless-drug scope: drug_development_stopped, drug_withdrawn

The workflow command uses --source=approvals, which means “collect approval evidence for eligible ownerless drugs.” It does not require clinical-trial evidence to also exist.

Indication extraction

This is where drug approvals connect to diseases. It runs as a separate IndicationWorkflow (37 steps) triggered after the drugs workflow.

Step 1: Structure labels (LLM)

GPT-4.1 reads raw label text and structures it into parsed sections, transforming free-text paragraphs into machine-readable fields. Eligible sources are FdaDatum, EmaApprovedDrug, PurplebookRecord, and ChinaApprovedDrug. CDE approvals prefer indications structured from the translated official label, with legacy indication fields retained only as fallbacks.

Step 2: Parse indications (LLM)

GPT-4.1 extracts individual Indication records from structured label text. Each indication captures:

• Disease name and subtypes
• Treatment lines (first-line, second-line, maintenance)
• Required biomarkers
• Indication type (treatment, diagnosis, prevention)
• Accelerated vs. full approval dates

Step 3: Post-process

Links extracted disease names to canonical Disease records using the same entity resolution cascade from the clinical trials pipeline. Links biomarker mentions to Biomarker records. Extracts therapeutic areas and updates the parent drug.

Domain context

A single drug approval can contain multiple indications. Pembrolizumab’s FDA label lists indications for NSCLC, melanoma, head and neck cancer, Hodgkin lymphoma, and many more — each with specific biomarker requirements and treatment-line restrictions. The indication extraction pipeline decomposes this into individual structured records.

Schedule and triggers

Component	Schedule	Job class
Full drugs workflow	Mon, Wed 00:00 UTC (cron) or manual via admin UI	DrugsWorkflowSchedulerJob (cron); WorkflowRunnerJob (manual)
Indication workflow	Auto-triggered after drugs workflow	WorkflowRunnerJob
FDA approval notifications	Sat 12:00 UTC	FdaApprovalNotificationsJob

Trigger manually:

# Full drugs pipeline
WorkflowRunnerJob.perform_now(workflow_type: "DrugsWorkflow")

# Just the indication extraction
WorkflowRunnerJob.perform_now(workflow_type: "IndicationWorkflow")

The ownership suggestion step is part of DrugsWorkflow and runs automatically as Build Ownership Suggestions after Indications: Update Drug Therapeutic Areas.

Key services

Service	Purpose
FdaService	Downloads and unzips FDA data files
Fda::LabelSync	Fetches label metadata from DailyMed
Fda::DailyMedClient	HTTP client for DailyMed API
DrugMerger	Merges duplicate drug records
OpenAiService	LLM calls for label structuring and indication parsing
OwnershipSuggestions::DrugSyncService	Builds and refreshes the ownerless-drug ownership suggestion queue

Key Thor tasks

Task	Purpose
regulatory:fda:download_and_extract	Download FDA drugs and labels
regulatory:ema:download_and_extract	Download EMA XLSX export
regulatory:cde:collect	Sync CDE records and collect, translate, and structure eligible official labels
regulatory:cde:sync	Sync chemical and marketed CDE records via API
regulatory:approvals:assemble	Create unified DrugApproval records
regulatory:matching:match_fda_to_bioloupe	Match FDA records to Bioloupe drugs
regulatory:matching:match_cde_to_bioloupe	Match chemical CDE records using English source fields
regulatory:indications:structure_labels	LLM-structure label text
regulatory:indications:parse	LLM-parse individual indications
regulatory:fda:fetch_labels	Fetch labels from DailyMed
regulatory:fda:fetch_boxed_warnings	Pull boxed warnings from SPL XML
regulatory:purplebook:sync	Sync Purple Book biological products
regulatory:companion_diagnostics:sync	Sync FDA companion diagnostics
clinical_trials:ownership_suggestions:sync	Build or refresh ownership suggestions for eligible ownerless drugs

Common problems

Symptom	Likely cause	Fix
FDA download returns 403	API key expired or rate limited	Update API key in the FDA Thor task
EMA XLSX structure changed	EMA updated their export format	Check column names in the EMA Thor task
Label fetch hangs	DailyMed API timeout	Restart with --skip-completed to resume
Duplicate DrugApprovals	Source key mismatch	Check source_type/source_id uniqueness
Innovation type is nil	New submission class code	Add mapping in approvals.thor

Next steps

• News and intelligence — How press releases and publications become structured intelligence
• Data model — Deep dive into the approval and indication tables