06 Llm Extraction And Verification

Inference Architecture

Status: CONVERGED DESIGN / PROVIDER ACTIVATION GATED. Runtime, lifecycle records, task/profile ownership, native structured output, prompt/source guards, retry-inclusive spend preflight, and a ten-case semantic corpus are implemented. Offline golden/negative evaluation passes. No paid provider-backed semantic evaluation has been run, so all three inference jobs remain off. Denoising state: converged (noise -> rough -> sharp -> converged) Updated: 2026-07-25

The one-sentence model

An asset names a task and supplies atomic source-grounded inputs; the inference runtime executes the active profile, records the attempt, validates the candidate, appends an acceptance decision, and returns only accepted output.

task + exact input
  -> request
  -> model attempt
  -> candidate output
  -> deterministic findings
  -> acceptance decision
  -> optional evidence projection

LLMs structure evidence. They do not become a source, curate data, or write canonical/query facts directly.

Settled decisions

1. Direct structured output is the default. One model attempt returns one complete candidate matching the task schema.
2. A request is atomic. One request has one whole-output decision. If fields need independent acceptance, split the task into independent requests.
3. Deterministic validation is not truth. It proves schema shape, exact quote support, and explicit invariants only.
4. Reuse requires acceptance. Provider success or schema validity alone is never cache eligibility.
5. What and how have separate identities. Task contracts define meaning; execution profiles define provider/model settings; routes select an active profile and dispatch policy.
6. Dagster orchestrates cohorts, not model rows. Postgres owns row-level lifecycle state. Do not create one Dagster partition per inference request.
7. External calls do not run inside domain write transactions. Read inputs, close the transaction, run inference, then open a short projection transaction.
8. Complexity must earn its way in through evals. Consensus, repeated sampling, LLM verifiers, agentic editing, and free-draft packaging are not default protocol features.

OpenAI’s Structured Outputs documentation makes the same important distinction we preserve: constrained output can guarantee shape, but it does not guarantee correct values. Its eval guidance recommends task-specific, continuously run evaluations before adding architectural complexity. Structured Outputs, Evaluation best practices

Public interface

Asset code uses only this level:

inputs = [InferenceInput(key="...", payload={...}, source_artifact_keys=(...))]
run = runtime.execute("soc_line_setting_decomposition", inputs)

for input_key, output in run.accepted.items():
    project(output)

Assets do not:

• construct provider wire requests;
• choose a vendor or model;
• parse vendor responses;
• write lifecycle tables;
• interpret ok=True as accepted;
• branch between fixtures and live providers.

The implementation lives in:

lib/llm/catalog.py      generic contract, profile, route, and input types
lib/llm/tasks.py        code-owned task contracts, profiles, and active routes
lib/llm/runtime.py      the public execute/poll/ingest behavior
lib/llm/validation.py   findings and decision policy
lib/llm/store.py        memory and Postgres lifecycle stores
lib/llm/request.py      internal provider-neutral wire types
lib/llm/providers/      vendor adapters only

Three things that change independently

Concept	Owns	Does not own
InferenceTask	task key, immutable contract revision, prompt, output schema, validators, decision policy	provider, model, batching
ExecutionProfile	provider, model, temperature, reasoning effort, token limit	task meaning, acceptance
TaskRoute	active contract revision, active profile, batch preference and threshold	prompt text, schema, provider wire format

Why this matters:

• A model retune changes a profile, not the task contract.
• A new output field changes the task contract, not merely the model.
• A stricter validator changes the acceptance-policy hash, so an old decision cannot be silently reused.
• An in-flight batch retains its exact contract revision and profile snapshot even if the active route changes.

The catalog retains immutable contract revisions while batches, replay, or provenance may reference them. Do not mutate a revision in place after it has produced attempts.

Four lifecycle records

Record	Grain	Identity	Mutable?
inference_requests	desired task over one exact input	task key + contract hash + input hash	no
inference_attempts	one model execution	request + profile hash + execution id + role + sample index + transport	output fields complete once; profile and pricing are retained snapshots
inference_findings	one validator observation on an attempt/path	attempt + validator revision + code + path	no
inference_decisions	one policy judgment on an attempt/path	immutable decision event	no; supersede

provenance.accepted_inference_results is a view over accepted decisions. There is no mutable result cache table.

Provider batch state is operational support in provenance.inference_batch_jobs; it is not a fifth semantic record.

Request versus attempt

This distinction prevents most lifecycle bugs:

same task contract + same exact input = same request
different model/profile/sample/run/transport = different attempt

The request key intentionally excludes provider and model. Therefore:

• an accepted production result may be reused after a model route changes;
• an evaluation or refresh run passes reuse_accepted=False to force a new attempt;
• accepted evaluation attempts, including repeated samples, never satisfy production reuse;
• changed validators/policy cannot reuse a decision with a different acceptance-policy hash.

Bounded HTTP/rate-limit retries are counted inside one attempt. A deliberate rerun, repeated sample, verifier call, or independent model vote is a separate attempt.

Acceptance

The runtime always follows this order:

1. Parse the provider candidate.
2. Validate it against the task’s JSON Schema.
3. Run task-declared deterministic validators.
4. Append all findings.
5. Apply the versioned decision policy.
6. Return the output only when the state is accepted.

Decision states are:

• accepted: the candidate passed its policy; only production attempts are eligible for production reuse and projection;
• rejected: retained for audit but unavailable to callers;
• needs_review: retained and unavailable until an explicit verifier/review policy supplies a new decision.

Current deterministic validators support:

• numeric sentinel rejection;
• percentage-range checks;
• exact source-quote substring checks;
• nested cited-item checks against named captured artifacts, including conditional gates such as “candidates are forbidden unless asset identity is verified”;
• rejection when a cited payload document names an artifact not declared by the request’s source_artifact_keys;
• per-request allowed-value checks that keep each output inside its declared facet scope;
• scalar/range consistency checks that reject missing, reversed, or ambiguous numeric bounds;
• literal claim/value and numeric-token occurrence inside that claim’s exact source quote;
• fill-missing-only protection so model output cannot replace already structured values;
• unique-item checks for repeated claims;
• JSON Schema validation for types, enums, required fields, nullability, and extra fields.

They cannot prove that a clinically plausible extracted value is true. A high-impact task that needs semantic checking must declare verifier_required and retain verifier decision references, or remain needs_review.

Atomicity rule

The request grain and acceptance grain are identical.

Example: if a task returns {line, setting, source_quote} and the quote fails, the entire candidate is rejected. We do not quietly keep line while discarding setting.

When partial acceptance becomes a real product need, split the task or introduce explicit candidate subpaths with their own decisions. Do not add ad hoc field filtering in an asset.

Sync and async execution

The same task contract and validation path serve both transports.

flowchart LR
  A["Asset cohort"] --> P{"Active route"}
  P -->|sync| S["Bounded concurrent attempts"]
  P -->|native batch| B["Persist attempts and submit"]
  B --> O["Sensor polls provider state"]
  O -->|complete| R["Dagster ingestion run"]
  S --> V["Validate and decide"]
  R --> V
  V --> D["Postgres lifecycle"]

For async batches:

1. The submitting run persists requests, pending attempts, and a local preparing batch record.
2. The provider returns a durable batch handle, which advances the record to submitted.
3. inference_batch_poll_sensor polls only external state.
4. On completion it yields a RunRequest with a stable run_key.
5. inference_batch_ingestion fetches, validates, decides, and persists results in an observable Dagster run.

The local preparing insert is an atomic submission claim. In Dagster, cohort identity is derived from the run id and owning asset, then combined with the task, profile, sorted request keys, role, and sample index. An op retry therefore finds the same claim even though it constructs a fresh runtime, while a deliberate re-execution gets a new run id. Input order cannot change cohort identity. Attempt role, sample index, and transport remain part of attempt identity, so evaluation samples cannot collapse into one batch.

Batch ingestion writes lifecycle records only. A batch-capable domain pipeline must model evidence projection as a subsequent asset run or explicit rematerialization that reads accepted results; the polling sensor and generic ingestion job never write domain evidence on its behalf. The current SOC route is synchronous, but its provider call and later projection are already isolated from deterministic claim extraction.

Sensors must not hide payload ingestion as sensor-side database writes. Dagster documents RunRequest.run_key as the deduplication mechanism for sensor-launched work. Dagster sensors

Use Dagster concurrency pools when provider or database limits need deployment-wide control. Dagster concurrency

Do not model every inference row as a dynamic partition. Dagster recommends keeping partition counts to roughly 100,000 or fewer; row state belongs in Postgres while Dagster owns meaningful cohorts/backfills. Dagster partitions

Expensive changes and discovery work

The lifecycle supports small, deliberate experiments without changing production semantics:

Need	Correct operation
Inspect a hard edge-case set	select stable request/input keys; call runtime.evaluate(...) so the attempts are isolated from production reuse
Compare two models	call runtime.evaluate(...) on the same requests with two named profiles
Sample one model N times	call runtime.evaluate(...) with distinct sample_index values
Refresh accepted historical rows after a retune	explicit production backfill/refresh cohort with reuse_accepted=False
Tighten a deterministic validator	bump validator revision/policy; old decisions are not reusable
Add or change an extracted field	create a new task contract revision and projection revision
Fix only evidence mapping	bump the projection revision; do not call the model again

An evaluation corpus should contain common cases, known failures, ambiguous cases, null/negative cases, and slice-specific high-impact examples. Promotion of a profile or protocol requires a task-level quality/cost/latency comparison, not anecdotes.

Implemented semantic corpus and activation gate

lib/llm/semantic_eval_cases.json contains ten revision-pinned cases across every active task: positive source-fixture cases, negative controls, prompt injection, fill-missing behavior, numeric operator/bound handling, requested-facet scope, downstream-biomarker confusion, ambiguous asset identity, and regimen-partner confounding. Each case retains its exact source reference and artifact key, a golden candidate, bounded semantic assertions, and known-invalid candidates where useful.

Default tests:

• refuse corpus drift from an active task revision;
• require coverage of every active task and the critical dimensions above;
• run real schemas, deterministic validators, decision policies, and semantic assertions over all golden candidates;
• prove known-invalid candidates fail with their expected guardrail findings;
• verify representative SOC and trial inputs still match checked-in source fixtures.

This is deterministic contract/eval infrastructure, not evidence that the model produces the goldens. Before any task route is enabled, run the opt-in paid test over all ten cases with the pinned profile and an explicit per-case ceiling:

RUN_LLM_SEMANTIC_EVALS=1 \
BIOLOUPE_LLM_EVAL_MAX_COST_USD_PER_CASE=<reviewed-ceiling> \
ANTHROPIC_API_KEY=... \
uv run pytest tests/test_llm_live_smoke.py -q -k semantic_eval

Record pass/fail, latency, token usage, actual cost, model/profile hashes, and reviewed failures. One provider connectivity smoke or one plausible answer is not a promotion gate. Anthropic likewise recommends multidimensional, representative, task-specific evaluation rather than a single aggregate score. Anthropic evaluation guidance

Deferred protocols

These are extension points, not implemented defaults:

• multiple models voting;
• the same model sampled repeatedly with consensus;
• producer then independent LLM verifier;
• free-form draft followed by structured packaging;
• agentic JSON editing tools.

Adopt one only when a representative eval shows a material gain that justifies extra cost, latency, failure modes, and provenance. Consensus itself is not truth; correlated models can agree on the same unsupported claim.

Failure semantics

Failure	Runtime behavior
sync call, batch poll, or result fetch timeout/rate limit	bounded transport retry inside the attempt/operation
batch-create timeout or lost response	no blind retry; persist submission_unknown for operator reconciliation
permanent provider exception	record failed attempt, then fail the Dagster run
malformed/non-object or truncated/content-filtered response	failed or incomplete attempt plus rejected decision
missing/unknown provider completion reason	incomplete_generation finding plus rejected decision
schema or deterministic validation failure	successful model attempt, findings, rejected decision
missing batch result/custom-id mismatch	record available outcomes, mark batch failed, fail ingestion run
failed/cancelled/expired batch	terminal operational state; no accepted result
unknown task/profile/revision	fail closed before provider execution
active profile has no reviewed price	fail closed before provider execution; cost must not silently become zero

Raw provider responses, parsed candidates, usage, cost, profile snapshot, pricing snapshot, Dagster run id, asset key, and errors are retained on attempts. Never reconstruct an accepted result from an ok flag.

Batch submission first persists a local preparing record. Submission is not blindly retried: a timed-out create request may already have been accepted by the provider. If the response is lost, the job becomes submission_unknown for operator reconciliation instead of silently creating a duplicate batch. A successful response attaches the provider handle and advances the local state to submitted.

Model, prompt, and cost accounting

The three active routes use task-sized profiles over pinned claude-sonnet-4-6:

Task	Active contract	Profile	Prompt/output limits	Default cohort/budget
soc_line_setting_decomposition	v3	soc_structured_extraction@v2	32,000 prompt bytes / 2,048 output tokens	5 / $2
missing_target_candidate_suggestion	v4	target_structured_extraction@v2	160,000 / 8,192	1 / $2
trial_eligibility_extract	v6	trial_eligibility_structured_extraction@v2	96,000 / 16,384	5 / $10

The Anthropic adapter owns a deliberate two-schema boundary. The immutable task schema is the local contract and is always checked in full after generation. A derived wire schema converts nullable type arrays, requires closed objects, enforces Anthropic’s optional/union complexity limits, and moves API-unsupported constraints into descriptions before sending output_config.format={type: json_schema, schema: ...}. This does not weaken local validation. Provider-enforced shape is only the first gate: refusals, token-limit stops, and missing/unknown completion reasons are quarantined before task validation. Anthropic documents the supported subset, SDK constraint transformation, complexity limits, and the fact that refusal or token exhaustion can prevent schema-conforming output. Anthropic structured outputs

Prompt inputs use canonical JSON rather than Python repr, XML assembled from untrusted text, or a free-form concatenation. Active prompts identify source documents as untrusted, tell the model to ignore embedded instructions and outside knowledge, define ambiguous domain fields, permit negative/empty output, and require exact source-bound quotes. Deterministic validators then enforce the parts software can prove; no prompt claim is treated as a safety boundary. This follows Anthropic’s current guidance to give clear context, explicit constraints, and examples/evals rather than relying on vague role prompting. Anthropic prompting guidance

Anthropic documents 4.6-and-later model IDs as fixed snapshots, not moving aliases. Reviewed prices are keyed by provider and model, never model name alone. Each attempt retains the exact reviewed input, cached-input, output, and batch-discount rates used for cost calculation, so later price-table changes cannot rewrite history. Anthropic model IDs, Anthropic pricing

Every production execution has a preflight hard ceiling derived from the rendered prompt bytes, schema, profile output limit, reviewed price, cohort size, and all configured synchronous retry attempts. It assumes no batch discount. Unknown pricing, missing output limits, oversized prompts, or a ceiling above max_cost_usd fail before dispatch. Default cohorts fit their default budgets even at each profile’s maximum allowed prompt and output size. llm_cost_cap remains a separate retrospective time-window check; it is not mislabeled as the dispatch budget.

Active generation profiles also pin a ten-minute request timeout. It is part of the profile hash and attempt snapshot rather than a hidden adapter constant. Transport retries remain inside one logical model attempt and retain their zero-based retry count even when all calls fail; the spend ceiling reserves every configured call because a lost timeout response can have unknown billing state. Batch create calls are still never blindly retried. Anthropic recommends bounded retries for transient failures and longer-lived streaming or batch operation for long requests. Anthropic API errors

SOC reference implementation

defs/evidence/soc.py demonstrates source-first optional enrichment:

1. evidence/soc_claim_tables is the single owner of the base claims, immutable projection relation, and effective view.
2. evidence/soc_claims reads only normalized rows explicitly marked projection_kind='authored_fixture_expectation' and writes deterministic approved-use review candidates; it has no LLM resource or llm kind.
3. evidence/soc_line_setting_gap_fill defaults enabled=false, accepts an exact approach-claim cohort, and caps each run with max_requests.
4. Eligible rows must be current deterministic claims, must have an immutable source artifact, and must lack a receipt for the active task/source/projection revision.
5. soc_line_setting_decomposition runs outside a domain write transaction.
6. A short transaction stores accepted output in an immutable projection row, including a valid zero-field result, so replay cannot pay twice.
7. evidence.soc_treatment_approach_effective applies only the active projection revision over the exact source artifact. Source-structured values always win.

This task is enrichment over a pre-existing candidate, not live label decomposition, curation, or clinical authority. Live openFDA rows are source sections and fail closed before candidate projection. A future governed regulatory approved-use task must split one exact compound section into zero or more atomic, quote-bound items with content-derived identities and immutable zero-result receipts. Even accepted items remain approved-use review candidates; they do not become SOC without independent professional-guideline evidence or explicit human promotion. See ADR-0011.

A reviewed correction still belongs in the append-only curation overlay described in 05-agentic-curation-and-overrides.md.

Clinical-trial eligibility reference implementation

defs/evidence/trial_eligibility.py demonstrates governed gap-fill after deterministic extraction:

1. evidence/trial_claims writes the conservative deterministic subset first.
2. provenance/legacy_llm_reuse_decisions marks each current trial/task not_required, reused, needs_fresh_llm, or rejected. Only needs_fresh_llm permits provider work.
3. evidence/trial_eligibility_gap_fill is optional, defaults enabled=false, accepts an explicit trial-key cohort, and caps each run with max_requests.
4. Task contract trial_eligibility_extract v6 can emit only requested facets, cites an exact quote and declared artifact, binds each claim to the deterministic inclusion/exclusion section that contains its quote, binds text and numeric values to that same quote, rejects duplicate claims, and represents numeric ranges with explicit minimum and maximum bounds.
5. Inference runs outside the database write transaction. A short transaction then projects only accepted outputs and stores the acceptance decision id.
6. An immutable projection receipt is written even for a valid zero-claim output, so replay cannot pay for or project the same source/task/revision twice.
7. Canonical eligibility reads claims only from the trial’s current source artifact; a new source snapshot cannot silently retain stale inferred evidence.

This policy proves schema, quote, request scope, and numeric consistency. It does not pretend to prove clinical truth. Disease and biomarker identity resolution remains a separate step, and fuzzy matches remain parked without accepted resolution evidence.

The explicit run config is intentional: Dagster positions asset config for manual runs, backfills, and debugging, while stable deployment behavior belongs in resources. The llm concurrency pool provides the deployment-wide provider throttle. Dagster asset configuration, Dagster concurrency

Missing-target review implementation

defs/resolution/target_suggestions.py demonstrates a different, deliberately non-publishing task shape:

1. Build a current cohort of canonical drugs with no deterministic target action.
2. Require an already verified publication-to-drug link and captured publication artifacts.
3. resolution/target_candidate_suggestions defaults enabled=false; an enabled run accepts an exact drug-key cohort, caps max_inputs, and runs in the llm concurrency pool.
4. Validate identity evidence against an exact known drug name and quote, then validate candidate citations against exact artifact text.
5. Keep every otherwise-valid candidate in needs_review.
6. Append immutable run/candidate receipts and a review-queue item.

The asset is not upstream of the canonical target or query graph, and its independent default-off gate prevents accidental calls even when selected directly. Review approval is a later, independent decision; the LLM cannot write a canonical drug-target action.

Change checklist

For a new or revised task:

1. Define an immutable task contract in tasks.py with source-grounded prompt, strict schema, validators, and decision policy.
2. Define an execution profile and route separately.
3. Keep each InferenceInput at the intended decision grain with artifact keys and locator.
4. Add golden, negative, malformed, source-quote, reuse, and policy-change tests.
5. Persist the acceptance decision and projection revision on any evidence fields created from the result.
6. Add a representative eval before changing model, protocol, consensus, or verifier policy.
7. Keep external calls outside database write transactions.

Rejected alternatives

• One mutable task matrix: conflates semantic contract, provider tuning, dispatch, and budget.
• One llm_calls cache/ledger table: conflates desired work, executions, validation, acceptance, and reuse.
• Schema-valid means accepted: confuses transport/shape guarantees with semantic policy.
• Model/provider in request identity: prevents fair comparisons and semantic reuse.
• Sensor-side ingestion: hides work, retries, and failures from Dagster runs.
• Agentic editing by default: adds state and cost before evals show a need.
• One Dagster partition per request: pushes row-level lifecycle into the wrong system.