Learn how a modern unified data platform on Databricks effectively reduces costs and streamlines data workflows for a leading global sporting retailer
The Results
{{stats-component}}
The Execution
The customer is a leading global sporting goods retailer with a footprint in 56 countries and selling products from over 20 brands.
The customer built their existing data platform on disparate cloud services (Amazon Redshift for data storage, Python/SQL for data processing, and Jenkins for workflow automation). The existing platform introduced many integration challenges that led to increased costs and reduced operational efficiencies.
ADA helped design, implement, and migrate their existing data to a new unified data platform on Databricks. We successfully resolved many technical challenges including handling and migrating diverse data sources, overcome limited support for converting Python code to PySpark using innovative solutions, and ensuring data alignment between the old platform and the new Databricks platform.
The project was executed seamlessly, preserving data integrity and security while avoiding disruptions to ongoing operations.
The Approach
Our goal is to establish a gold layer within the Databricks system, utilizing data from the silver layer for business intelligence reporting. Our approach to achieve this involves:
- Data Source Analysis: Understand current data sources and associated codes for a clear migration foundation.
- Code Transformation with Optimization and Quality Checks: Convert Python/SQL to Pyspark/Spark SQL, adding data quality checks for reliable data integrity and optimize it for space and speed.
- Data Validation and Comparison: Verify Databricks data against Redshift data, ensuring seamless alignment.
- Data Orchestration: Schedule jobs using Databricks workflows, implementing monitoring and alerts for a reliable workflow.
- Documentation and Knowledge Transfer: Document the migration process for reference and conduct knowledge transfer sessions for enhanced team adoption.
