A metadata-driven foundation for Truveta Data

By Srini Burugapalli, Chief  Data Architect at Truveta

Hospitals produce varied patient data including structured electronic health records (EHRs), labs, medications, billing; unstructured clinical notes; imaging; physiological monitoring signals; wearables; genomic data. Researchers and clinicians need these  data to be clean, consistent, comparable across institutions, and traceable to their source. But every hospital runs different software, uses different field names for the same concepts, and delivers data in different formats.

Truveta ingests data from 50+ source types*, including leading US health systems and claims sources, representing more than 140 million patients. At that scale, trustworthiness isn’t a data-volume problem. It’s a structural one and getting it right is what makes the research and clinical AI built on top of these data something that people can actually trust.

Scale context

Truveta Data includes 50+ source types, 5,500+ schema elements, 100K+ fields, 13,000+ versioned transforms, and petabyte-scale data that is processed daily as it lands.

We’ve run into every one of the challenges below ourselves. This is the architecture we built to solve them, written for data engineers, architects, and technical leads evaluating how to build or scale a metadata-driven data platform.

Part I — Problems every platform encounters at scale

1. Schema drift and versioning chaos
   Source schemas change constantly. Pipelines built against an older schema silently produce wrong results, with no queryable history of what the schema looked like at any given time, and no clean way to roll back.
2. Scattered transform logic 
   A transform maps source data to a target model. That logic ends up spread across SQL files, dbt models, Spark jobs, and scripts across different repos — understanding one mapping means navigating several codebases.
3. Data quality as an afterthought
   Checks get bolted on after pipelines are built, disconnected from the transforms they validate. When one fails, it’s often unclear which pipeline stage caused it.
4. Lineage breaks at tool boundaries
   Lineage tools track their own system (dbt tracks dbt, Spark tracks Spark). Cross-tool lineage needs every tool to emit to a shared standard  and breaks the moment one doesn’t. In healthcare, “where did this data come from and what was done to it” is a regulatory requirement, not a nice-to-have.
5. Slow, error-prone source onboarding
   Every new source has a unique schema. Manually mapping it, then writing and validating transforms doesn’t scale past a handful of sources, let alone 50+.
6. Orchestration as a second maintenance surface
   DAGs (Airflow, Dagster) are authored separately from the data logic they run. The two drift apart over time, and keeping them in sync gets harder as pipeline count grows.
7. Platform lock-in for modelers
   People who define what data should look like get forced to also know how and where it runs (SQL dialect, compute platform, orchestrator) — so a platform migration means retraining that shouldn’t be necessary.
8. Multi-model output, no shared lineage
   Clinical data ships as relational tables, documents, graphs, and customer extracts. A typical stack needs a separate toolchain per output type, with no shared lineage between them.

Part II — The metadata-driven solution

Core principle: the catalog is the system of record. Truveta’s catalog holds schema definitions, transform logic, lineage, quality expectations, dependency order, and output model specs — versioned, queryable, diffable, and rollback-capable, much like Git for data infrastructure. Pipeline code isn’t handwritten; it’s generated from the catalog. Change the catalog, and downstream pipelines regenerate.

1. Schema drift → versioned schema management
   Every schema element has full version history in the catalog, with Git-style branching, PRs, and merges, plus visual diffs between any two versions. When a health system changes its EHR schema, the catalog updates and pipelines regenerate automatically while the old version stays queryable.
2. Transform logic → versioned metadata, not files
   Transform intent (what a mapping should produce) is separated from execution artifacts (the generated SQL/Spark). You can query for every transform that produces a given element, like medication, along with its full history, and regenerate code any time the catalog definition changes. More than 13,000 versioned transforms are managed this way.
3. Data quality → co-defined with transforms
   Quality rules live in the catalog alongside the transforms they validate, so they evolve together automatically. A key invariant is enforced at every stage boundary: the record count entering a layer equals the count passed through plus the count quarantined. A profiler tracks this continuously and pinpoints exactly which stage transition introduced a discrepancy. On top of that: a data profiler (nulls, distributions, cardinality, outliers), a rule-based/ML quality engine, data classification against catalog-defined value sets, and monitoring/alert thresholds. All of it is authored as catalog artifacts and versioned with the schema they apply to.
4. Lineage → continuous, model-agnostic
   Lineage is a property of the catalog’s transform definitions, not of whichever execution framework ran them. Since all four output models derive from catalog-managed transforms, the source-to-extract trace stays complete, a prerequisite for trustworthy clinical AI.
5. New source onboarding → AI-assisted schema mapping
   A schema-mapping AI agent reads an incoming source schema, generates the field-level mapping to the Truveta Data spec, and produces a gap analysis of what was specified vs. delivered. The output is a catalog update, not hand-written SQL — and parameterized transform templates then generate the health-system-specific transforms, enforcing harmony across sources.
6. Orchestration → generated, not authored
   DAGs are derived from catalog transform metadata rather than maintained by hand; the catalog’s dependency graph determines merge order and referential integrity across entities. Truveta runs on Conductor OSS — a pragmatic organizational choice, not an architectural one. Because transform logic lives in the catalog, AI agents can generate DAGs for any orchestrator (Conductor, Dagster, Airflow, Prefect) from the same definition. Switching orchestrators means regenerating a layer, not rewriting transforms.
7. Platform lock-in → modeling separated from execution
   Modelers define what a field should contain and how it maps from source data — not Spark code or platform-specific SQL. A platform migration (e.g., Databricks to Snowflake) means regenerating execution artifacts, not retraining modelers or rewriting transform definitions.
8. Multi-model output → one catalog definition, shared lineage
   Four fit-for-purpose models are derived from the same catalog definitions:

• Relational — normalized schema for cohort analytics and BI/SQL tooling.
• Document — optimized for rapid retrieval of longitudinal patient journeys against complex phenotypes, including unstructured notes.
• Graph — patient journey analysis, provider networks, clinical pathway inference, temporal sequencing, treatment-patient clustering, off-label use.
• Customer extracts — de-identified, schema-versioned, consumed in Spark/Python/R.

Shared lineage across all four comes for free from the catalog, instead of being rebuilt per output type.

Part III — The AI layer

At 50+ source schemas and 20+ ontologies, there are too many codes, terms, and formats to map by hand with fixed rules. AI shows up throughout the platform to handle that scale:

• Terminology services map local codes and more than 64 million free-text terms to 20+ standard ontologies (ICD-10, SNOMED-CT, LOINC, RxNorm, CPT, etc.) using models trained on the platform’s actual data distribution.
• NLP extraction agents pull structured facts (diagnoses, medications, procedures, device use, labs, observations) out of unstructured notes at scale.
• Annotation infrastructure generates labeled training data from Truveta’s 7 billion clinical notes via active learning, with a closed feedback loop from production performance back into training data quality.
• Schema mapping engine AI-assists mapping new source schemas to canonical catalog schemas.
• AI-assisted catalog authoring lets developers describe a catalog change or inconsistency in natural language and have an agent generate the update directly.
• Pipeline code generation — AI agents generate Spark pipeline code, validation notebooks, and orchestration DAGs from catalog metadata, and regenerate them when a target changes.

Part IV — The pipeline in practice

Data flows through four stages, then fans out from the canonical Truveta Data Model (TDM) into fit-for-purpose outputs:

Source Data → Landing (Historical + Active) → Staging TDM → Common TDM

↓

Relational | Document | Graph | Customer Extracts

Landing layer: deliberately two-tier, because health systems routinely resend overlapping data. The historical layer is an immutable append of every row ever received — a full audit trail that lets the active layer be fully reconstructed if a bad file needs retracting. The active layer holds the latest deduplicated record per catalog-defined identity key, with SCD tracking applied per entity type as defined in the catalog — never hardcoded.

Pre-production validation: before a transform ships, catalog-generated validation scripts check inserts/updates, duplicates, and referential integrity. Production only runs pre-validated transforms.

Synthetic data for end-to-end testing: real patient data can’t be used for development or CI/CD, so synthetic data stands in across the entire pipeline, from a single transform to a full run through all four output models and de-identification.  — That synthetic data is generated from catalog schema constraints plus the profiler’s real production statistics (null rates, cardinality, categorical frequencies), so it behaves like real data. Because synthetic data, transforms, and distributions all derive from the same catalog version, there’s no drift between test and production.

This metadata-driven architecture is also the foundation of the broader Truveta platform, transforming fragmented clinical records into normalized, traceable data, reproducible evidence, and real-time intelligence. A new peer-reviewed publication in JAMIA Open describes how Truveta combines health system governance, AI-enabled harmonization, daily updates, de-identification, and rigorous quality controls to support population-scale clinical research and real-world evidence generation.

The bottom line

Schema drift, scattered transform logic, quality-as-afterthought, lineage gaps, slow onboarding, and dual maintenance surfaces all share one root cause: logic and intent get entangled with execution artifacts.

The metadata-driven approach separates them. The catalog holds the intent. AI and code generation produce the artifacts. Change the catalog, and everything downstream regenerates. That’s what makes the platform extensible to new orchestrators, output models, and sources without proportional growth in maintenance burden, and making lineage, quality, and governance tractable at petabyte scale across dozens of health systems.

*A “source type” is a distinct data-producing system at a health system or third party, not the health system itself — a single health system can contribute multiple source types (e.g., separate EHR systems and internal data warehouses each have their own schema, delivery format, and mapping requirements).