Data Source Onboarding as a Service
Standardization and automation of enterprise data ingestion
Client: Package Delivery Service Provider, USA
6 months
to MVP, first business unit fully migrated
BYOD
source teams ingest their own data
Batch + stream
one onboarding process for both
Challenge
Autonomous IT groups had left the data siloed and duplicated, with no standard way in: 200+ JMS sources, 1000+ Teradata tables, 30+ Oracle data marts and more. Onboarding had to stop depending on IT and start working the same way for batch and streaming alike.
Solution
An Azure-based data platform on a Bring Your Own Data approach: source teams ingest into a central lake and warehouse themselves. Streaming and batch both supported, with harmonization, automated ingestion and metadata capture.
Implementation
- 01Streaming through on-prem Kafka, Azure Event Hub, Functions and Spark Streaming (Kappa architecture)
- 02Batch through Azure Data Factory and Databricks
- 03As-a-service configuration for ingestion and harmonization
- 04Centralized ADLS Gen2 and Delta Lake, organized by domain
- 05Azure Purview for metadata, glossary and data catalog
Business impact
MVP delivered in 6 months
First business unit fully migrated with minimal IT involvement
One ingestion process for batch and streaming data
Less duplication and lower storage costs across departments
Technology stack
Have a complex system to build or modernize?
Tell us what must change. We will bring the relevant domain and engineering leads into the conversation.