Cloudera Data Platform Migration
On-prem big data infrastructure moved to Azure PaaS
Client: Oil & Gas Industry Leader, USA
16
months to full migration, against 18 planned
4
months to MVP
8+ TB
of data, 55+ ingestion components, 25+ analytics products
Challenge
A complex on-prem Cloudera platform had to reach Azure within 18 months, with minimal refactoring and no disruption to the business. It carried 55+ ingestion components, 25+ analytics products and 8+ TB of varied data, tied to legacy technologies like Oozie and HDFS.
Solution
A phased Azure PaaS migration run MVP-first, with reusable accelerators doing the repetitive work of moving ingestion components and analytics products. Hub-and-spoke architecture, event-based pipeline execution, and custom tooling for automation, validation and monitoring.
Implementation
- 01Discovery and complexity assessment of the existing systems
- 02Migration PoCs and MVP on Azure HDInsight, ADLS Gen2 and Data Factory
- 03Accelerators for ingestion generation, pipeline migration, CI/CD and data validation
- 04Event-driven architecture, with ExpressRoute for on-prem connectivity
- 05Hub-and-spoke resource separation for cost and security isolation
Business impact
MVP delivered in 4 months, full migration in 16
The on-prem platform decommissioned
New capabilities along the way: granular billing, smart monitoring, on-demand compute, data lineage
Automated ingestion and pipeline migration, faster and more consistent than by hand
Technology stack
Have a complex system to build or modernize?
Tell us what must change. We will bring the relevant domain and engineering leads into the conversation.