Matt Adams

Databricks LakeFlow Connect is currently in Gated Public Preview. As this is the case, there isn’t a ton of documentation around setting up CI/CD processes using Databricks Asset Bundles (DAB). Fortunately the setup is pretty straight forward, and I found it to already be supported by DABs. Ok let’s get... [Read More]

Traditional data warehousing often relies on auto-incrementing IDs as primary keys, leading to dependencies that can hinder data freshness, particularly in streaming environments. This article explores an alternative approach: utilizing deterministic primary keys to improve data loading efficiency and achieve faster data insights. [Read More]

We traded in our clunky, nightly batch processing for a sleek, real-time streaming data machine on Azure Databricks. By cleverly combining a hybrid approach with some nifty key generation tricks, we supercharged our data freshness and dramatically slashed reprocessing costs. [Read More]