Healthcare data engagements

Governed data delivery for a global pharmaceutical brand

The challenge

The client’s commercial and partner teams needed custom data views built to strict specifications, and data had to move between Databricks and Snowflake with column names, data types and precision exactly preserved.

What we did

• Lead the data engineering team on the account, owning sprint planning, estimation and delivery
• Built custom views in Azure Databricks to partner specifications and delivered them to production
• Built cross-platform data sharing from Databricks to Snowflake with explicit data contracts, so types and precision never drift
• Built configuration-driven ingestion with automated QA rules for new data
• Built data marts that feed the client’s AI and reporting teams

The outcome

Partner and analytics teams receive governed, specification-exact data from one platform, shared directly across Databricks and Snowflake.

Specialty pharmacy data feeds for pharmaceutical manufacturers

The challenge

Pharmaceutical manufacturers depend on scheduled feeds of inventory, dispense and regulatory programme data. The feeds ran on legacy stored procedures and needed to move to a modern platform with identical output.

What we did

• Built and shipped daily, weekly and monthly data feeds to production on Azure Databricks
• Migrated legacy stored procedures to parameterised Databricks notebooks, with output reconciled against the legacy files
• Applied industry data standards and formatting rules required by manufacturers
• Took ownership of handed-over feeds and trained engineers to support them

The outcome

Manufacturers receive reliable, on-schedule feeds from a modern, maintainable platform in production.

A unified data platform for a multi-hospital network

The challenge

Patient and operational data arrived from several hospitals in different formats. It had to be combined, standardised and masked before analysts could use it, without exposing sensitive information.

What we did

• Built ingestion and masking workflows for data from every hospital in the network
• Developed dbt models and YAML source configurations to standardise transformation across sources
• Built and managed Apache Iceberg tables on Databricks and Snowflake
• Ran daily production processing for the whole network

The outcome

One consistent, masked data layer that analysts across the network can query, with transformation logic defined once in dbt instead of repeated per hospital.

A metadata-driven ingestion framework

The challenge

Every new client data source needed a hand-written pipeline. Onboarding was slow, latency was high and schema changes in source data regularly broke production.

What we did

• Replaced one-off pipelines with a metadata-driven framework on Azure and Databricks, so new sources are added through configuration
• Optimised the ingestion path to reduce end-to-end latency
• Added schema validation and automated monitoring at the point of ingestion

The outcome

Ingestion latency fell by about 70%, onboarding a new client or source became about 90% faster, and production data issues dropped by about 80%.

Automating operational data for a manufacturer

The challenge

Production and operational data was collected and combined by hand every day. Most of the team’s time went into compiling reports instead of acting on them.

What we did

• Mapped the existing workflows and found where data was re-entered or copied
• Built a centralised system that collects and processes operational data automatically
• Delivered management reporting and process monitoring on top of the new data

The outcome

The share of daily work done manually fell from about 80% to about 10%, and managers got consistent, timely reports on production activity.

sproc-to-lakehouse: migrating T-SQL stored procedures to Databricks, with proof

The challenge

Moving stored procedures to Databricks is often done by translating the SQL line by line. The translation compiles, but some of it quietly changes the results or fails at run time, and nobody can say which procedures are safe until they are checked by hand.

What we did

• Built an open-source tool that parses T-SQL and flags every construct as automatic, automatic with a note, or needing a human decision
• Generated a dbt project and Databricks notebooks for each procedure, with a report of what changed and why
• Checked every generated statement against Apache Spark: 19 of 19 statements and 10 of 10 dbt models were valid
• Ran each converted procedure and the original on the same seeded data, on SQL Server 2022 and on Spark with Delta Lake, and compared the results row by row

The outcome

All six fully converted procedures returned identical data on both platforms. With the tool’s fixes turned off, only two of six did; the other four failed on Spark. The remaining six procedures are flagged with the exact steps a person must convert. All procedures and data are synthetic, and every figure comes from running the code.

Explore the interactive results →