ANFIMAU | Analytical data exposure

Overview

Our client, a leading provider of legal information services, offers legal departments and law firms web-based analytics services with reporting tools for department performance assessment, expense analysis, and legal budget analysis. In addition it provides access to aggregated data from more than thousands of legal departments and tens of thousands of law firms around the world to compare your own legal departments data with the best-performing ones and identify areas for optimizations.
As part of a new business initiative, the customer aims to enable a Data Exposure solution as an extension to an existing web-based analytical service to provide clients capabilities to access data using their own BI tools or export data into own systems via automated data processes.

Challenge

The initiative for a new business offering arises from the need to meet specific client requests for customized reporting needs which cannot be accommodated by the existing web-based analytical reporting modules.

Data exposure solution should be multi-tenant addressing strong security requirements for data storage and isolation, support infrastructure costs segregation per client (tenant) and utilize Azure PaaS technologies to be compliant with customer technology strategy.

Solution should support two selective data exposure patterns to end-clients:

BI Pattern: Highly performant data exposure via protocols supported by most popular BI tools used by clients (such as PowerBI, Tableau, Qlik) and optimized for custom reporting needs.
Integration Pattern: Low-cost data exposure via HTTPs-based protocol for clients to connect and extract data into own reporting solutions via external automated data pipelines.

Enablement of new data exposure channels should not affect performance of existing web-based analytical service while ensuring data consistency and will be offered to clients as add-ons to the main offerings.

Provisioning and management of data exposure capability for clients should be fully automated with minimum overhead to customer IT.

Approach

Engagement started with a discovery phase that included detailed design preparation considering solution FRs, NFRs, identified ASRs followed by PoCs execution to assess performance, project infrastructure costs and other non-functional aspects.

Architecture approach prepared as part of the discovery phase along with PoC results and phased delivery plan was socialized to the customer to get the sign-off and kick-off delivery phase.

The delivered solution is based on Azure PaaS technologies and leverages following approaches:

Multi-tenant architecture & security

Multi-tenant architecture with dedicated infrastructure resources per tenant to address strong security & customer SLA requirements and support costs chargebacks per tenant.

Secure data connectivity & replication

Data exposure tier that supports secure connectivity by clients leveraging AAD & secure data transfer protocols, provides eventually consistent read only data storages storing replicas optimized for data exchange and incremental data processing and leveraging technologies suitable for required data exposure patterns.

Analytical product migration strategy

Recurrent patterns for analytical products migration steps were identified which resulted into creation of custom accelerator to automate migration routine for data pipelines of analytical products (e.g. generation of adf pipelines to wrap oozie workflows execution, auto-refactoring of source code to be compliant with HDInsight, etc.).

Incremental data pipeline management

Data pipeline templates with support of incremental data processing to organize data movement to target data storage for each data exposure pattern.

Data schema versioning and compatibility

Reliable data schema enforcement with backward compatibility support.

Dynamic data processing activation

Event-based triggering of data processing pipelines based on new data availability.

Administrative control & infrastructure management

Control plane for the customer admins to enable end-to-end tenant management and full automation of infrastructure resources provisioning and data pipeline deployments based on templates required for client selected data exposure pattern.