Situation
Our client is a global technology firm with a large BI analytical platform comprised of transactional systems (ERP, CRM) and custom applications. Maintaining data quality was an issue due to inconsistency and a varied standardization of KPIs and metrics in silos, across different systems. This fragmentation contributed to data redundancies.
The modern data platform increased operational efficiency and improved data accuracy and reliability.
Other challenges included using common master data for reporting, maintaining up-to-date records, implementing live data streaming, and managing complex ETL jobs and metadata.
Since multiple systems extracted the same information to cater to different business groups, duplication of reports was common.
Action
We conducted an analysis of the Cloudera environment, identified dependencies, planned the migration steps and timeline. To address cost, scalability, and maintenance concerns, we migrated data from our client’s Cloudera-based Hadoop infrastructure to Azure.
For an efficient data transfer in a big data environment, we used tools for data integration and migration, mapping to fit the new Azure SQL Server schema. Lakehouse architecture was utilized to establish layers for landing, history, current, and archival data.
We implemented a solution using Databricks with Azure Data Lake and Azure SQL Server, employing an SQL-based framework for data cleansing and transformation.
To ensure master data consistency, we identified an "owner" system for each dimension table across all transaction systems. KPIs/metrics were consolidated with standardization, resulting in a single source of truth for reporting across the global organization.
Our solutions have given our client an increase in operational efficiency and improved data accuracy and reliability for faster decision-making and cost savings.
Results
- Improved data accuracy and reliability in reports.
- Acceleration in implementation of processes due to high reusability rate for components.
- Infrastructure and maintenance savings as a result of reduced hardware resource usage.
- Increase in operational efficiency due to reduction in data duplication.
Reducing hardware resource usage led to savings in infrastructure and maintenance.
When data duplication was reduced, operational efficiency increased.
Migration of data to Azure addressed cost, scalability, and maintenance issues.
Reducing hardware resource usage led to savings in infrastructure and maintenance.
When data duplication was reduced, operational efficiency increased.
Migration of data to Azure addressed cost, scalability, and maintenance issues.
Reducing hardware resource usage led to savings in infrastructure and maintenance.
-
30%
Reduction in data loading time -
-
40%
Higher reusability rate for components -
-
20%
Cost savings in infrastructure and maintenance