Merciven
Work · Data engineering

Hospital data platform

Ingestion, masking and dbt transformation for six US hospitals on Databricks and Snowflake.

Organisation
Benttella Consulting LLC (US, remote)
My role
Data Engineer
When
2026
Stack
Databricks, Snowflake, dbt, PySpark

Problem statement

Each hospital sent data in its own format and schedule, and patient identifiers had to be masked before anyone downstream could use the data.

Objectives & goals

  • Onboard a new hospital by configuration, not new code
  • Mask personal data before it leaves the raw layer
  • Give analysts consistent, tested models
  • Catch bad data before it reaches reports

Process

  1. DiscoverProfiled each hospital's feeds and their quirks.
  2. DefineConfig tables for sources and QA rules; data contracts for outputs.
  3. BuildConfig-driven bronze ingestion, masking, silver merges, dbt gold models on Iceberg.
  4. ProveQA gates that stop the job, contract checks and reconciliation against legacy output.

Who it's for

Analytics team

Needs the same numbers from every hospital.

Compliance

Needs proof personal data is masked.

Challenges

  • Six hospitals, six formats
  • Patient data under strict access rules
  • Silent type drift breaking downstream tables

What I built

Config-driven ingestion

One generic pipeline reads a sources table.

PII masking

Hashed keys, initials and masked emails before silver.

QA rules

A failed error rule stops the job before bad data spreads.

Data contracts

Block type drift such as decimal(38,2) to decimal(38,6).

Flow

  1. Hospital feeds
  2. Config-driven bronze
  3. QA rules
  4. Silver MERGE
  5. Mask PII
  6. dbt gold on Iceberg
  7. Contract check
  8. Analytics

Screens

Architecture of the open-source Databricks demo that mirrors this platform, on synthetic data.
Architecture of the open-source Databricks demo that mirrors this platform, on synthetic data.

Results

6hospitals
13,000+records processed a day
~70%lower ingestion latency on metadata-driven pipelines
~90%faster client onboarding

Tools & languages

  • Databricks
  • Snowflake
  • dbt
  • PySpark
  • Apache Iceberg
  • Azure

The code

A complete Databricks demo on synthetic data: config-driven ingestion, QA rules, masking, data contracts and reconciliation.

The repository is private. I'm happy to share it with a hiring team or walk you through it on a call.

Request a code walkthrough · Read the other healthcare data engagements

Samples are written with synthetic data. No client code or client data is used.