27 Slice Discovery Search

Pipeline Slice: Discovery and Search

Status: SHARP CORE VERTICAL. Canonical entity lookup, real-document lexical search, facets, owner-row provenance, and SQL recipes are implemented. Additional document families join only through their owning slices. Denoising state: sharp (noise -> rough -> sharp -> converged) Updated: 2026-07-23

Purpose

Let read-only SQL agents find:

• a canonical entity by preferred name, alias, title, or source identifier;
• a real publication or news document;
• a source-backed text chunk mentioning accepted entities;
• unresolved or review-parked mention claims;
• the owning q_* row and provenance receipt behind every result.

Discovery is a terminal query projection, not a source pipeline, identity owner, extraction framework, or search service.

The Implemented Boundary

flowchart LR
  Names["Canonical identity names and aliases"]
  Trials["Canonical trials"]
  Approvals["Canonical approvals"]
  Publications["Normalized/canonical publications"]
  News["Normalized/canonical news"]
  EntitySearch["q_search_entities"]
  Documents["q_discovery_content_items"]
  Chunks["q_discovery_chunks"]
  Evidence["q_search_evidence"]
  Mentions["q_discovery_mentions"]
  Facets["q_discovery_facets"]
  Projection["q_search_projection"]
  Provenance["q_provenance"]

  Names --> EntitySearch
  Trials --> EntitySearch
  Approvals --> EntitySearch
  Publications --> Documents
  News --> Documents
  Documents --> Chunks --> Evidence
  News --> Mentions
  Documents --> Facets
  Documents --> Projection
  Chunks --> Projection
  EntitySearch -. owner row .-> Provenance
  Documents -. owner row .-> Provenance
  Chunks -. owner row .-> Provenance
  Mentions -. owner row .-> Provenance

There are no sources_discovery, normalized.discovery_*, evidence.discovery_*, resolution.discovery_*, or canonical.discovery_* tables or assets.

Why This Shape

The legacy Data Gov discovery subsystem had content items, bodies, chunks, mentions, links, and facets. Its useful intent was source-neutral retrieval. Its accidental shape was a second data pipeline that could duplicate source documents, extraction, resolution, and identity.

The greenfield correction is:

legacy intent: search across evidence
legacy accident: copy evidence into a second mini-domain
greenfield design: project the real owner records into terminal SQL search views

This passes the deletion test:

• deleting discovery never deletes a publication, article, mention claim, or canonical entity;
• rerunning discovery never calls a source or model;
• discovery cannot invent an entity key;
• source corrections flow through their owner and appear on the next view query.

Source Inclusion Contract

A document family joins discovery only when its owning slice provides:

1. a stable document key;
2. source text or owner-produced chunks;
3. an immutable source artifact key;
4. a provenance-bearing owner q_* row;
5. accepted entity keys with an explicit scope, when available.

Current coverage:

Owner	Searchable content	Entity-link scope	Provenance owner
publications	title + captured abstract; one abstract chunk	document	q_publications
news	headline + owner-produced body chunks	document	q_news_articles / q_news_search_chunks

Not admitted yet:

• regulatory labels, because exact FDA source sections now exist in normalized.regulatory_label_* but the regulatory owner has not published a stable provenance-bearing document/chunk q_* contract; normalized.label_documents remains embedded multi-authority document identity only;
• SEC filing prose, because the commercial owner retains exact Inline XBRL text but has not published a human-readable document/chunk contract;
• arbitrary fixture text or fabricated “representative” documents.

Absence is explicit. Discovery does not manufacture text to make a source look complete.

Entity Lookup

q_search_entities reads:

• resolution.entity_names populated by the drug, disease, biomarker, target, technology, organisation, endpoint, and adverse-event owners;
• canonical trial titles and registry identifiers;
• canonical approval product names and application numbers.

Each row contains:

• resolved_entity_key;
• entity_type;
• search_name and normalized_name;
• name_kind and is_primary;
• entity_state;
• source_q_model_name and source_q_row_key;
• the owner provenance key when exposed.

Document and mention frequency do not determine whether an entity is searchable. A canonical entity remains discoverable before any article mentions it.

Document And Chunk Semantics

q_discovery_content_items is one real source-owned document. It carries accepted entity keys aggregated by its owner facts.

q_discovery_chunks is one lexical retrieval unit:

• a publication abstract is one source-abstract-v1 chunk;
• news reuses the normalized owner chunk and records news-paragraph-480-v1;
• character offsets and immutable artifact keys remain explicit;
• entity_link_scope='document' states that current accepted keys apply to the document, not necessarily to every sentence.

The explicit scope prevents a dangerous retrieval shortcut: an article-level entity key must not be misrepresented as a span-local extraction.

Mention Semantics

q_discovery_mentions projects real owner-produced mention claims. News is currently the only document owner with an implemented mention-span grain.

It preserves the owner states:

• linked — exactly one accepted owner identity;
• review — ambiguous or otherwise parked;
• unresolved — no accepted candidate.

Discovery does not rerun a hardcoded dictionary and does not collapse review into unresolved.

Query Models

Model	Grain	Purpose
q_search_entities	canonical entity name/alias	exact and trigram-assisted entity lookup
q_discovery_content_items	real source document	document listing, full-text title/summary search, owner pivot
q_discovery_chunks	owner-produced lexical chunk	passage retrieval with artifact locator
q_discovery_mentions	owner mention claim	linked/review/unresolved inspection
q_search_evidence	searchable lexical chunk	source-neutral evidence retrieval
q_search_projection	document or chunk row	uniform retrieval shape
q_discovery_facets	facet assignment	source, language, and accepted entity-type filtering
q_query_recipes	checked-in SQL example	in-database usage guidance

q_query_recipes is a static VALUES view. Recipes are documentation, not canonical mutable state, so no table or Python seeding asset exists.

Provenance Contract

Every search row except a recipe carries:

source_q_model_name + source_q_row_key

The blocking search_provenance gate requires:

1. the named query.q_* relation exists;
2. q_provenance contains a receipt for that exact model/row tuple.

Every discovery asset therefore declares the owner q_* relations it can name plus query/q_provenance as Dagster dependencies (directly or through another discovery projection). Canonical data dependencies alone are insufficient because the blocking gate dereferences the published owner surfaces.

provenance_key is retained as a convenience when the owner exposes one, but the model/row tuple is authoritative. This avoids pretending that source artifact keys and provenance bundle UUIDs are the same namespace.

Search Technique

PostgreSQL is the first search engine:

• tsvector / tsquery for lexical evidence retrieval;
• pg_trgm for typo-tolerant entity names;
• typed arrays and facets for entity/source filtering.

Primary references:

• PostgreSQL full-text search: https://www.postgresql.org/docs/current/textsearch.html
• PostgreSQL pg_trgm: https://www.postgresql.org/docs/current/pgtrgm.html

Semantic embeddings are not in the current contract. No null embedding column, fake model metadata, or unindexed vector placeholder is published. Add vectors only after a named retrieval evaluation shows lexical/trigram search misses an important workflow, then record model, dimensions, corpus revision, cost, and recall evidence.

Quality Gates

Blocking:

• schema_contract — live columns and SQL types match the checked-in contract;
• search_provenance — every row reaches a real owner receipt;
• query_recipe_contract — recipes are SELECT-only and reference existing q_* views;
• owner resolution policy — only accepted owner keys enter resolved_entity_keys.

Structural:

• provenance_coverage remains attached to every query asset as required by the shared query-layer convention; search views are retrieval affordances, so owner-row provenance is enforced by search_provenance rather than duplicate receipts for every projected row.

Warnings to add when volume justifies them:

• stale source documents;
• high owner-level unresolved/review rates;
• measured lexical retrieval regressions;
• query latency/index regressions.

Tests And Implementation Owners

• assets: src/bioloupe_data/defs/query/discovery.py;
• SQL: src/bioloupe_data/sql/query/discovery/;
• contracts: src/bioloupe_data/lib/checks/contracts_schema/discovery__*.yaml;
• reusable gate: lib.checks.make_search_provenance_check;
• slice proof: tests/test_slice_discovery.py;
• shared gate proof: tests/gates/test_core_gates.py.

The DB smoke publishes all eight views twice and verifies source/type/scope invariants. Offline tests fail if the synthetic discovery pipeline, fake dictionary/reference union, or embedding placeholders return.

Completion Boundary

The core slice is implemented for current real text owners. Future work belongs to owner slices:

• labels must publish normalized versioned text before discovery indexes it;
• commercial filings must publish a readable prose/chunk contract before discovery indexes them;
• vectors require a retrieval benchmark, not architectural enthusiasm.

Discovery itself should remain small as source coverage grows.