03 Repository Blueprint

Repository Blueprint

Status: IMPLEMENTED CORE / RUNTIME GAPS VISIBLE. Package layering, resources, loaded definitions, Alembic revision ownership, and one protected query family are implemented. Source partitions and corpus-wide protected publication remain incomplete. Denoising state: sharp (noise -> rough -> sharp -> converged) Updated: 2026-07-24

Purpose

Define the buildable shape of the bioloupe-data repository: package layout, where Dagster definitions live, where SQL assets live, how resources/secrets are configured, and the local/CI commands. This is surrounding architecture, not slice content.

Project standard

Adopt the modern Dagster project standard produced by create-dagster / the dg CLI: a src/-layout Python package whose code location is a single top-level Definitions object, with an autoloaded defs/ folder. (https://docs.dagster.io/guides/build/projects/creating-projects, https://docs.dagster.io/guides/deploy/code-locations/managing-code-locations-with-definitions)

• One code location to start. A code location is a module exposing a top-level Definitions, loaded in its own process; a deployment may hold several, but we begin with one and split only when dependency isolation demands it. (https://docs.dagster.io/deployment/code-locations/managing-code-locations-with-definitions)
• uv for environment + lockfile. dg is optimized for uv; nothing else ties us to it. (https://github.com/dagster-io/dagster/discussions/30285)
• Organize defs/ by concept (pipeline layer), not by technology. Dagster explicitly supports organizing by data-processing concept to give context to engineers unfamiliar with the underlying tech — which matches our layered source->q_* architecture. (https://docs.dagster.io/guides/build/projects/project-structure/organizing-dagster-projects)

Directory tree (decided)

bioloupe-data/
├── pyproject.toml            # deps + [tool.dg] config; package metadata
├── alembic.ini               # credential-free schema migration config
├── uv.lock
├── README.md
├── docs/                     # this spec corpus
├── src/
│   └── bioloupe_data/
│       ├── __init__.py
│       ├── definitions.py    # top-level Definitions (autoloads defs/, merges pythonic)
│       ├── migrations/       # immutable Alembic revisions; no Dagster definitions
│       ├── defs/             # AUTOLOADED layer assets, organized by concept
│       │   ├── __init__.py
│       │   ├── sources/      # source acquisition (one subpackage per source)
│       │   ├── normalized/   # source-faithful normalized records
│       │   ├── evidence/     # evidence-claim extraction (deterministic + LLM)
│       │   ├── provenance/   # source artifacts, extraction runs, assertion links
│       │   ├── resolution/   # entity-resolution candidate assets
│       │   ├── curation/     # observed curation override/review store + sensors
│       │   ├── canonical/    # canonical knowledge-graph assets
│       │   └── query/        # q_* SQL read-model asset definitions
│       ├── sql/
│       │   ├── provenance/   # DDL/view helpers for provenance tables where SQL-owned
│       │   └── query/        # checked-in SQL text for q_* models
│       └── lib/              # shared library code (NO definitions live here)
│           ├── __init__.py
│           ├── resources/    # ConfigurableResource classes (db, http, llm)
│           ├── llm/          # inference catalog/runtime/store, validation, cost, provider adapters; NO Dagster definitions
│           ├── partitions/   # shared PartitionsDefinitions
│           ├── checks/       # reusable asset-check factories
│           ├── provenance/   # hash, locator, PROV-shaped receipt helpers
│           ├── sources/      # source-specific capture adapters, value types, repositories
│           ├── sql/          # SQL rendering/execution helpers
│           └── types/        # domain value types + schema helpers
└── tests/
    └── __init__.py

Sensor placement is decided: sensors are co-located with the external state they observe — reviewed-curation sensors live in defs/curation/sensors.py, and inference batch-polling sensors live in defs/provenance/sensors.py because they observe persisted batch provenance state. The inference sensor launches a normal ingestion job; it does not ingest payloads as a sensor side effect.

Rationale for defs/ vs lib/: dg projects separate autoloaded definitions (defs/) from reusable library code (types/decorators/factories) so definitions are discovered without an “import circus”. Dagster’s current project-structure guide shows a src/<project>/definitions.py plus defs/ layout, and explicitly allows organizing projects by data-processing concept when that gives engineers more context than organizing purely by technology. (https://docs.dagster.io/guides/build/projects/project-structure/organizing-dagster-projects)

Slice Module Layout

The implemented slice set confirms the layer-first defs/ structure, with slice/source subpackages inside each layer to keep ownership navigable.

defs/
  sources/
    ctgov/ aact/ chictr/ regulatory/ publications/ news/ organisations/ public_health/ sec/ vocabularies/
  normalized/
    trials.py approvals.py publications.py news.py organisations.py epidemiology.py commercial.py drug_identity.py
  evidence/
    trials.py approvals.py publications.py news.py soc.py organisations.py epidemiology.py commercial.py
  provenance/
    source_artifacts.py extraction_runs.py inference_lifecycle.py assertion_links.py bundles.py sensors.py
  resolution/
    subjects.py drugs.py interventions.py diseases.py biomarkers.py organisations.py endpoints.py
  curation/
    observed_decisions.py effective_overrides.py review_queue.py conflicts.py sensors.py
  canonical/
    trials.py approvals.py publications.py news.py soc.py organisations.py drug_identity.py epidemiology.py commercial.py development_programs.py discovery.py
  query/
    trials.py approvals.py publications.py news.py soc.py organisations.py drug_identity.py epidemiology.py commercial.py development_programs.py discovery.py support.py

sql/query/
  trials/
    q_trials.sql q_trial_interventions.sql q_trial_endpoints.sql q_trial_outcomes.sql q_trial_eligibility.sql
  approvals/
    q_drug_approvals.sql q_approval_indications.sql q_label_evidence.sql q_regulatory_product_protections.sql
  publications/
    q_publication_evidence.sql q_publication_trial_links.sql q_publication_outcomes.sql
  news/
    q_news_events.sql q_news_entity_mentions.sql q_intelligence_events.sql
  drug_identity/
    q_drugs.sql q_drug_aliases.sql q_drug_external_ids.sql q_drug_brands.sql q_drug_regimens.sql
  development_programs/
    q_development_programs.sql q_development_program_state_history.sql q_development_program_milestones.sql
    q_development_program_evidence.sql q_emerging_clinical_evidence.sql
  epidemiology/
    q_epidemiology_observations.sql q_epidemiology_series.sql q_disease_epidemiology_summary.sql
  commercial/
    q_product_sales.sql q_product_commercial_summary.sql
  shared/
    q_catalog.sql q_provenance.sql

sql/provenance/
  source_artifacts.sql extraction_runs.sql inference_lifecycle.sql assertion_provenance_links.sql

Rules:

• defs/<layer>/... owns Dagster definitions only. Shared parsing, SQL rendering, resources, and check factories live in lib/.
• defs/provenance/ owns shared provenance assets whose grain crosses slices: immutable source artifacts, extraction runs, assertion-provenance links, and generated bundle helpers. Slice-local evidence schemas still live in defs/evidence/<slice>.py.
• Slice files may import shared value types from lib/types/ and receipt helpers from lib/provenance/, but cross-slice business concepts must also be documented in 20-pipeline-families.md.
• Cross-slice synthesis such as therapeutic development programs lives in defs/canonical/ and consumes canonical owner assets. It never reads query.q_* back into the pipeline and does not introduce a new graph layer.
• defs/query/support.py is for downstream-support read models such as optional forecasting readiness; it must not contain downstream app state or API contracts.
• Python modules use normal importable filenames (trials.py, not trials.sql.py). SQL text is kept as adjacent, reviewable .sql files under sql/query/.

definitions.py (decided)

The top-level object autoloads the defs/ folder and merges any hand-written Pythonic definitions and shared resources via Definitions.merge. (https://docs.dagster.io/guides/build/projects/project-structure/combining-components-with-pythonic-definitions, https://docs.dagster.io/api/dagster/definitions)

src/bioloupe_data/definitions.py

from pathlib import Path
import dagster as dg
from bioloupe_data.lib.resources import default_resources

@dg.definitions
def defs() -> dg.Definitions:
    layer_defs = dg.load_from_defs_folder(path_within_project=Path(__file__).parent)
    base = dg.Definitions(resources=default_resources())
    return dg.Definitions.merge(layer_defs, base)

A Definitions object carries assets, asset_checks, schedules, sensors, jobs, and resources. (https://docs.dagster.io/api/dagster/definitions)

Resources and secrets (decided)

• Resources are typed ConfigurableResource objects (Postgres, shared HTTP client config, source-specific capture resources, inference runtime), constructed centrally in lib/resources/ and bound in Definitions(resources=...). Source-specific resources live with their source module under lib/sources/<family>/resource.py and are exported through lib/resources/ only when assets depend on them.
• Secrets and connection details come from environment variables, never hardcoded — including per-provider LLM keys (OPENAI_API_KEY, ANTHROPIC_API_KEY, TOGETHER_API_KEY, GEMINI_API_KEY/Google credentials, and DEEPSEEK_API_KEY if the DeepSeek adapter is enabled). Use Dagster EnvVar for launch-time secret/config binding; do not use os.getenv in Dagster definition modules or default resource constructors for source mode selection. Dagster notes that EnvVar resolves at run launch while os.getenv resolves when the code location loads. (https://docs.dagster.io/guides/operate/configuration/using-environment-variables-and-secrets)
• Stable resource keys are shared by assets, checks, and sensors:
  • postgres: transactional/query database access for assets, checks, provenance ledgers, and sensor cursor reads.
  • source_http: low-level HTTP client config shared by source-acquisition resources.
  • ctgov_source: ClinicalTrials.gov source-capture resource; owns fixture/live mode, CTGov API config, pagination, and CTGov adapter selection.
  • chictr_source: ChiCTR scrape-artifact resource; owns deterministic fixture vs explicit scraper-output import and never pretends ChiCTR has a generic live API.
  • epidemiology_source: official epidemiology source-capture resource; owns fixture/live mode, dataset identity, paging, live smoke bounds, and dataset adapter selection.
  • sec_source: SEC source-capture resource shared by filer and commercial filing assets; owns fixture/live mode, configured CIKs/official filing URLs, paging/bounds, and SEC adapter selection.
  • fmp_source: Financial Modeling Prep company-profile resource; owns fixture/live/import mode, configured symbols, provider credentials, and live smoke bounds. It does not own news or deal events.
  • regulatory_source: FDA/openFDA/Orange Book/Purple Book/EMA/KEGG/CDE source-capture resource; owns fixture/live mode, regulatory source adapter selection, live bounds, and FDA supplement capture.
  • authority_source: NCIt/RxNorm/MeSH authority source-capture resource; owns fixture/live mode, authority HTTP adapters, max_live_records, and missing-only/refresh lookup policy for derived-demand authority snapshots.
  • target_authority_source: HGNC/ChEMBL plus project-catalog source-capture resource; owns fixture/live mode, the live lookup bound, and missing-only/refresh selection for drug-derived target demand.
  • clinical_vocabulary_source: versioned endpoint-catalog and official NCI CTCAE source-capture resource; owns fixture/live mode, exact workbook capture/parsing, and optional bounded-live smoke.
  • publication_source: PubMed, Crossref, and conference source-capture resource; owns fixture/live/import mode, configured source windows, free official HTTP adapters, and bounded-live metadata.
  • news_source: newswire/provider-export source-capture resource; owns deterministic fixture vs explicit operator import. It does not claim a generic live API for licensed feeds.
  • regulatory_label_source: FDA SPL/openFDA label-document source resource; owns fixture/live/import mode, configured application/search windows, exact document capture, and bounded-live metadata.
  • legacy_data_gov_source: disabled-by-default migration-cache resource; owns explicit enablement, max_import_records, the legacy URL supplied through run config, and database-enforced read-only sessions. It is not a primary-source mode.
  • llm: ConfigurableResource that binds lib.llm.runtime.InferenceRuntime to the Postgres lifecycle store; evidence/resolution assets and inference batch jobs request tasks through this resource, never provider SDKs directly.
  • curation_store: read-only access to the external curation decision store for observation/checks; Dagster does not write, approve, or reject curation decisions.
• Source-specific resources depend on shared resources through Dagster resource dependencies, e.g. ctgov_source depends on source_http instead of each asset accepting the low-level HTTP resource. Live/fixture selection belongs to the source resource’s run config, not to asset-body if env_flag: fetch_live else: read_fixture branches. Dagster supports configuring resources at launch time and testing ConfigurableResources directly. (https://docs.dagster.io/guides/build/external-resources/configuring-resources, https://docs.dagster.io/guides/build/external-resources/testing-configurable-resources)
• New live-capable source families add one deep source resource when a real variation exists; do not add resource keys speculatively before assets depend on them.
• Anti-pattern (enforced): no heavy module-level imports or model loads in asset modules — they inflate the code-location process memory even when assets are idle. (https://u11d.com/blog/scaling-dagster-kubernetes-multi-code-locations/)

SQL asset implementation (decided)

q_* read-model assets live in defs/query/ as Dagster assets whose compute kind is SQL and whose body executes checked-in SQL text from src/bioloupe_data/sql/query/ (see 07-sql-query-layer.md). Start with plain Dagster assets, not dbt. Reconsider dbt only if model count, dependency compilation, and warehouse portability become real pressure; do not introduce dbt merely to create views.

The implementation shape should be boring:

1. defs/query/<family>.py declares individual assets or one non-subsettable multi-asset when the family is a consistency unit, and attaches owners, group, kinds/tags, dependencies, publication policy, and checks.
2. Each model loads one .sql file by stable package path and renders only safe compile-time identifiers/config values. In-place families use the small SQL lifecycle helpers; protected families hand all model SELECTs to the generation publisher.
3. SQL assets also apply COMMENT ON TABLE / COMMENT ON COLUMN statements or equivalent checked-in comment metadata so generated q_catalog has a database-native source of descriptions.
4. lib/sql/ owns the tiny execution/rendering helpers: read SQL file, parameterize approved identifiers, run inside a transaction, attach row-count/index/refresh metadata, apply comments, and fail loudly on unsafe template variables.
5. lib/checks/ owns reusable check factories; query assets attach schema_contract and provenance_coverage checks by default.
6. lib/provenance/ owns small deterministic helpers for source-artifact hashing, immutable locator validation, semantic fact_path construction, and provenance-bundle ids. It must not contain Dagster definitions.
7. lib/publication/ owns the deep transaction module for candidate freezing, physical contract enforcement, atomic stable-view promotion, idempotent retry, and retained-generation rollback. Domain validation remains in the owning slice module; clinical trials owns its manifest, cross-model candidate gates, publish entrypoint, and rollback entrypoint in lib/trials/query_publication.py.

Postgres materialized views store query results physically and must be refreshed explicitly, but that is a performance choice rather than a last-known-good mechanism. See 07 for the implemented generation-backed stable-view protocol.

flowchart LR
  PyDef["defs/query/trials.py\nDagster asset definition"]
  SqlText["sql/query/trials/q_trials.sql\nreviewable SQL"]
  TrialPolicy["lib/trials/query_publication\nfamily manifest + domain gates"]
  Publisher["lib/publication\ncandidate + atomic promotion"]
  Checks["lib/checks\nschema + provenance + owner gates"]
  Db["Postgres query.q_trials\nstable view"]

  PyDef --> TrialPolicy
  SqlText --> TrialPolicy --> Publisher --> Db
  Checks --> TrialPolicy

Query asset declaration contract

Every q_* asset definition must declare:

• asset key: ["query", "<q_model_name>"];
• group: query;
• owner slice;
• upstream canonical/evidence dependencies;
• stable relation kind plus physical backing: view, materialized_view, table, or generation-backed stable view;
• publication family/mode and whether partial materialization is forbidden;
• primary grain and primary key/natural key expression;
• contract version and deprecation status;
• attached checks: at minimum schema_contract and provenance_coverage, plus source_artifact_immutability where the query model exposes document/payload-derived facts;
• automation gate: downstream publication only after upstream blocking checks pass.

Dagster asset metadata, tags, owners, group names, and kinds are the repo-visible way to make the asset graph legible in the UI as well as in code. (https://docs.dagster.io/guides/build/assets/metadata-and-tags/)

Local dev and CI commands (decided baseline)

uv sync --extra dev           # install locked runtime and development dependencies
make dev                      # run Dagster UI + daemon locally
make check                    # lint + types + tests + definition validation
make verify-bootstrap         # execute the complete graph in a disposable blank database
make migration-check          # validate one linear Alembic revision graph
make migrate                  # upgrade an already-versioned database to head

The Make targets are the repository contract; they wrap the pinned uv environment and the underlying dg commands so local and CI execution stay identical.

Operational Boundaries

• Keep one code location until an observed dependency or deployment-isolation requirement justifies a split. Source families are not separate code locations merely because they are separate domains.
• A hot materialized view may use REFRESH MATERIALIZED VIEW CONCURRENTLY only after its contract and database checks prove a qualifying unique index that covers every row.