ANFIMAU | Data landscape consolidation

Overview

Our client, a market analytics firm, faced a fragmented data technological landscape across existing brand analysis products which resulted to:
- Lack of technology standardization and central platforms that could be used by new products to be rapidly built.
- Lack of foundational data management capabilities to ensure data quality and data reuse across products.
As a result, the client aims to come up with the new technology strategy for managing data and technology more effectively to decrease costs of delivery and time to market for new market analysis products and services.

Challenge

Products collect, process and store survey and non-survey data from the same type of source systems (org partners): survey engines, sampling engines, panel management systems, custom reference and master systems, external data providers. Such redundant data extractions and storage systems on organization level lead to costs overheads and inefficiencies.
Products leverage technologies for data collection, processing and storage they prefer which leads to lack of reuse, increased engineering efforts and maintenance costs on organization level.
There is lack of data reusability and data trust across products as well as lack of data discoverability for data exploration and analytical needs across products.
Inconsistent data management & security leading to operational & security risks.

All these challenges increase the demand for relevant software capabilities and new solutions to maintain IT/OT software applications within system operation and to cope with the increasing demand for change.

Approach

To address customer challenges the proposal was to build data platform leveraging Hub-and-Spoke architecture pattern and Azure PaaS technologies with a goal to:

Unified data source consolidation

Act as single source of company data (survey data, reference data, master data, sales, media and other data assets) segregated by data domains for all the new data products.

Streamlined data collection and utilization

Reduce costs and complexity involved in data collection thus reducing total time to market for new data analytical initiatives and increasing organizational capabilities reuse.

Data landscape cost optimization

Reduce overall cost of data landscape by:

Reducing redundant data extraction by multiple applications.
Reducing redundant storage for same raw data across multiple products.
Reducing complexity for data engineering across the organization.

Engineering efficiency and acceleration

Reduce engineering efforts and accelerate delivery of new analytical data products in a governed way through unified technology stack, reusable platform assets like data pipeline templates, IaaC modules, CICD pipeline templates, etc.

Automated data governance enablement

Enable data management through automated data governance processes and centralized Data Catalog containing metadata of company data estate

Security framework establishment

Establish security framework through automated security policies, auditing, monitoring, defined access models and security checklist / processes to follow

Data exploration & analysis facilitation

Provide data exploration capabilities for platform users for analysis and hypothesis verification needs

Delivery of the data platform was based on Agile practices and focused only on capabilities required for identified high-priority business and platform use-cases.

Efficient data management and cost reduction

Significant reduction in redundant data extraction and data storage, reducing costs, increasing efficiency & data reusability

Enhanced data engineering efficiency

Streamlined data engineering across company reducing 25% to deliver new analytical data products

Creation of unified platform-level data products

Established multiple platform-level data products with consolidated company data assets for various data domains used by analytical products, power users and advanced analytics

Rapid analytical product innovation

Accelerated delivery of 3 brand new analytical products leveraging reusable data platform assets and unified technology

As-a-service platform for analytical products

Introduction of as-a-service platform capabilities for analytical data product needs with support of API-based and event-based integrations

Infrastructure and DataOps enhancement

Enhanced monitoring, capacity management & chargebacks, infrastructure security & standardization through Infrastructure-as-a-Code and DataOps implementation

Data exploration environment satisfaction

Satisfaction of data analysts and other power users through data exploratory environments capability

Advanced data management and security

Improved data management across company through automated metadata capturing, data discoverability via Data Catalog, self-service data access, governed business glossaries and enhanced data security based on company policies