Role: Technical Lead | Dates: 2022 - 2024 | Status: Completed
Technologies: Azure Databricks, Apache Spark, PySpark, Python, SQL, Delta Lake, Azure DevOps, Git, CI/CD, Databricks Asset Bundles
Spearheaded the technical leadership for a team of 12 engineers on a critical initiative to modernize the enterprise data platform. Identified significant deficiencies in reporting availability due to legacy ETL performance and quantified the business impact. Developed and presented the comprehensive solution design, ROI analysis, budget forecasting (multi-million dollar investment), and implementation plan to executive leadership, securing buy-off and funding. Key activities included:
- Converting ~600 legacy code packages (1k-20k LOC each) to modern PySpark/Databricks equivalents.
- Establishing CI/CD pipelines in Azure DevOps using Databricks Asset Bundles (DABs) for automated testing and deployment.
- Creating standardized Databricks Asset Bundle templates to ensure consistency and accelerate project setup across the team.
- Onboarding and training over 100 developers on the new platform, tools, and best practices within the first year.
Outcomes:
- Reduced nightly ETL processing time from 8 hours to 2 hours (4x improvement).
- Improved critical reporting SLA compliance from 5% to 99%+.
- Enabled broad platform adoption through the successful onboarding and training of 100+ developers within the first year.
- Recipient of the 2024 PacificSource Enterprise Innovation Award for this work.