Matt Adams

Often when working with Unity Catalog, especially with managed tables, the underlying blob storage structure can feel a bit opaque. Understanding storage consumption at the blob level is crucial for cost analysis, identifying unexpectedly large tables, tracking storage growth patterns, and debugging storage issues. This guide provides a practical approach... [Read More]

As teams adopt Databricks Asset Bundles (DABs) for managing projects, ensuring consistency and accelerating setup becomes crucial. Instead of copying and pasting configurations, you can create DAB Templates. These templates allow you to define a standard project structure and configuration, while prompting the user for specific details (like project names,... [Read More]

Traditional data warehousing often relies on auto-incrementing IDs as primary keys, leading to dependencies that can hinder data freshness, particularly in streaming environments. This article explores an alternative approach: utilizing deterministic primary keys to improve data loading efficiency and achieve faster data insights. [Read More]

We traded in our clunky, nightly batch processing for a sleek, real-time streaming data machine on Azure Databricks. By cleverly combining a hybrid approach with some nifty key generation tricks, we supercharged our data freshness and dramatically slashed reprocessing costs. [Read More]