China Drug Trials pipeline

China Drug Trials (chinadrugtrials.org.cn) is the official Chinese drug clinical trial registry. Unlike AACT, it does not expose a stable public JSON API. Unlike ChiCTR, the useful downstream responses are not trial detail HTML pages alone: the site supports a browser-gated session bootstrap, a form-posted list search, an Excel export for list rows, and a Word export for trial details.

This document captures the verified request flow, the implementation shape that fits the existing clinical-trials pipeline, and the open questions to close before production rollout.

What was verified

These requests were tested live on June 24, 2026 from this repo environment.

1. Plain HTTP bootstrap is not enough

Direct HTTP to:

GET /clinicaltrials.searchlist.dhtml?reg_no=&indication=&case_no=&drugs_name=&drugs_type=&appliers=&communities=&researchers=&agencies=&state=

returns anti-bot responses when called without a real browser session. The site sets cookies such as:

• FSSBBIl1UgzbN7N80S
• FSSBBIl1UgzbN7N80T
• token

but the first plain HTTP response is not a cleared session. Browser JavaScript must run first.

2. Browser bootstrap + cookie handoff works

Using BrowserManager / Ferrum to load the bootstrap URL, wait for the page, and then read cookies from the browser session produced a reusable cookie header. That cookie header was then used successfully for direct HTTP POSTs.

Verified cookie names from live browser bootstrap:

• FSSBBIl1UgzbN7N80S
• FSSBBIl1UgzbN7N80T
• token

3. The list POST works with the browser-issued cookies

Verified request:

POST /clinicaltrials.searchlist.dhtml

with form fields such as:

• agencies
• appliers
• case_no
• ckm_index
• communities
• currentpage
• drugs_name
• drugs_type
• id
• indication
• keywords
• reg_no
• researchers
• rule=CTR
• secondLevel
• sort
• sort2
• state

Observed result:

• HTTP 200
• HTML response
• pagination present
• 20 CTR ids visible on page 1 for the unfiltered full-list request

4. The XLS export works with the same search payload

Verified request:

POST /clinicaltrials.searchlist.dhtml?_export=xls

using the same search payload as the list request.

Observed result:

• HTTP 200
• application/vnd.ms-excel
• Excel XML workbook
• 20 CTR ids present in the sample export, matching the HTML page-size sample

This makes XLS useful for list-level summary extraction and for validating list parsing.

5. The detail DOC export works with only id

Verified request:

POST /clinicaltrials.searchlistdetail.dhtml?_export=doc

with form body:

• id=<opaque row id>

Observed result:

• HTTP 200
• application/msword
• WordprocessingML payload
• contains the target trial registration number

This is materially better than relying on the HTML detail page because the DOC export appears to need only id, not ckm_index or the full search context.

Recommended ingestion architecture

High-level shape

Use a hybrid flow:

1. Browser bootstrap only for session clearance and cookie acquisition.
2. Direct HTTP POSTs for list pagination, XLS export, and DOC detail export.
3. Raw response persistence before parsing.
4. Import into clinical_trials and related tables through a source-specific parser/importer pair.

This matches the existing repo patterns:

• ChiCTR: browser-heavy collection, structured import
• CDE: browser bootstrap for cookies, then HTTP requests

Why this is the best fit

• It avoids brittle browser DOM scraping for every page.
• It keeps anti-bot handling isolated to one bootstrap step.
• It uses the cleanest available payload for each layer:
  • HTML or XLS for list pages
  • DOC export for trial details
• It supports deterministic retries and checkpointing.

Proposed collection flow

Step 1. Bootstrap session with browser

Open:

GET /clinicaltrials.searchlist.dhtml?reg_no=&indication=&case_no=&drugs_name=&drugs_type=&appliers=&communities=&researchers=&agencies=&state=

via BrowserManager.

Wait for the page to stabilize, then extract cookies from the browser context.

Persist in memory:

• cookie header
• bootstrap timestamp
• last successful bootstrap URL

Step 2. Fetch list pages by POST

Primary request:

POST /clinicaltrials.searchlist.dhtml

Primary baseline body for the full registry:

agencies=
appliers=
case_no=
ckm_index=
communities=
currentpage=1
drugs_name=
drugs_type=
id=
indication=
keywords=
reg_no=
researchers=
rule=CTR
secondLevel=0
sort=desc
sort2=
state=

For filtered runs, populate fields such as:

• keywords
• reg_no
• indication
• drugs_name
• appliers
• researchers
• state

Step 3. Parse list rows and pagination

From the HTML list page, capture:

• id — opaque trial row key
• ckm_index — row index from the element name attribute
• reg_no — trial identifier like CTR20241055
• row-level summary fields visible on the page
• total pages
• total records
• current page

Optional secondary capture from XLS:

• row summary columns
• row count per search/page

The HTML remains the primary row-discovery source because XLS does not expose the opaque id.

Step 4. Fetch details by DOC export

For each row discovered on the list page:

POST /clinicaltrials.searchlistdetail.dhtml?_export=doc

body:

id=<opaque row id>

Persist raw WordprocessingML and parse detail fields from that payload.

Step 5. Retry and refresh session

If any list/XLS/DOC response looks like anti-bot HTML, missing expected markers, or returns an unexpected content type:

1. discard current cookie header
2. acquire a fresh browser session
3. retry the failed request once with the fresh cookies

If the second attempt fails, surface the failure and checkpoint progress.

What to save

Raw artifacts

Persist raw source responses before parsing.

Recommended storage layout:

• silver/china_drug_trials/{run_id}/lists/page_{n}.html
• silver/china_drug_trials/{run_id}/exports/page_{n}.xls.xml
• silver/china_drug_trials/{run_id}/details/{reg_no}__{id}.doc.xml
• silver/china_drug_trials/{run_id}/summary_{run_id}.json

Why keep all three:

• HTML list is the source of row id
• XLS is a clean list-level validation/export format
• DOC is the preferred detail payload

Parsed checkpoint metadata

Checkpoint state should include:

• run_id
• bootstrap timestamp
• current cookie freshness timestamp
• current search context
• current page number
• total pages snapshot
• fetched row ids
• fetched registration numbers
• failures with error type and retry count

Import and dedupe strategy

Registry identifier strategy

The official registry uses identifiers like CTR20241055.

That does not collide with:

• AACT / ClinicalTrials.gov: NCT...
• ChiCTR: ChiCTR...

So the existing unique clinical_trials.nct_id constraint can still serve as the primary no-duplication guard across sources, as long as we preserve the official CTR... registration number exactly.

New collection source

A new collection_source value should be added for this registry, separate from aact and chictr.

Recommended value:

• china_drug_trials

This will require follow-on changes in:

• ClinicalTrial enum
• any source-filtered workflows and queries
• ownership suggestion enums or source filters if trial evidence should include this source

Import model

Follow the ChiCTR pattern:

1. parse raw detail payload into a structured hash
2. upsert clinical_trials first
3. populate related tables second
4. preserve expensive downstream fields such as llm_data during updates

Duplicate policy

Within this source:

• dedupe on reg_no / nct_id
• when the same reg_no reappears in later runs, update the main trial record and refresh source-derived related payload as needed

Across sources:

• do not merge with ChiCTR or AACT automatically
• treat separate registry ids as separate records unless a future reconciliation layer is explicitly designed

Translation strategy

The source content is largely Chinese. Translation should not happen during collection.

Recommended approach:

1. collect and persist raw Chinese first
2. import source fields in raw/original form
3. run translation as a separate downstream enrichment step

This mirrors the CDE pattern and keeps collection deterministic.

What to translate

Translate only fields needed for downstream English-facing processing, for example:

• titles
• indications
• sponsor / institution names when needed for review UX
• inclusion / exclusion criteria
• outcome descriptions
• arm descriptions

What to preserve verbatim

Always preserve raw Chinese source text for:

• provenance
• re-translation
• parser debugging
• future extraction improvements

Suggested storage pattern

Store both:

• raw source text
• translated normalized text

in source payload or dedicated translated metadata, rather than overwriting the original.

Expected service layout

Recommended new namespace:

• app/services/china_drug_trials/

Suggested classes:

• session_bootstrapper.rb — browser bootstrap and cookie refresh
• client.rb — POST list / XLS / DOC requests using current cookies
• list_parser.rb — parse HTML list pages, row ids, pagination
• detail_parser.rb — parse DOC WordprocessingML detail payload
• import_service.rb — DB import logic
• weekly_scraper.rb or collector.rb — orchestration, checkpointing, retries
• summary_manager.rb — run summary persistence

Implementation order

Phase 1: Collector proof

Build a minimal collector that can:

1. bootstrap cookies via browser
2. POST page 1 full-list HTML
3. extract 20 row ids
4. fetch one DOC detail export by id
5. save raw artifacts locally

Phase 2: Pagination and raw archive

Add:

• multi-page crawl
• checkpointing
• anti-bot refresh on expiry
• S3 upload of raw artifacts

Phase 3: Parser and importer

Add:

• DOC parser
• trial hash mapper
• clinical trial upsert
• related table import

Phase 4: Translation and downstream workflows

Add:

• translation job for source text
• sponsor-linking support
• study plan / eligibility downstream compatibility review

Open questions

These still need confirmation in code:

1. Whether a row id remains valid only within the current browser-cleared session.
2. Whether the DOC export always works with only id, or occasionally needs additional hidden fields.
3. Whether XLS can export beyond the current HTML page-size when the search result is larger.
4. Which detail fields are present in DOC export but absent from HTML, and vice versa.
5. Whether there are registry-specific fields that should map to existing clinical_trials columns or remain source payload only.

Current recommendation

Use:

• browser bootstrap for session acquisition
• HTML list POST as the authoritative row-discovery source
• XLS export as secondary list validation / optional summary import aid
• DOC export as the primary detail payload

Do not build this as a pure browser scraper unless the HTTP flows become unstable. The verified browser-bootstrap-plus-HTTP approach is cleaner, faster, and fits the repo’s existing China-source patterns.