Medallion Architecture from CDC History Tables

When done, you will have:

• A bronze layer provided by the upstream Lakebase Change Data Feed history tables (input to this recipe)
• A silver layer with deduplicated, current-state materialized views for each entity
• A gold layer with business aggregations and metrics as materialized views
• A scheduled Lakeflow Declarative Pipeline refreshing silver and gold layers incrementally
• All layers queryable as Unity Catalog tables via SQL, Spark, BI tools, and Genie

Prerequisites

This template builds a Lakeflow Declarative Pipeline on top of existing Lakebase Change Data Feed (CDF) history tables. Verify these Databricks workspace features are enabled before starting.

• Lakeflow Declarative Pipelines (serverless) available. Run databricks pipelines list --profile <PROFILE> and confirm the command succeeds (an empty list is fine). A permission or not enabled error means Lakeflow Pipelines is not available to this identity.
• Unity Catalog access with a writable destination schema. Run databricks catalogs list --profile <PROFILE> and confirm your destination catalog is listed. You will need USE_CATALOG on the catalog and USE_SCHEMA + CREATE_TABLE on the destination schema to publish silver and gold tables.
• Bronze CDC history tables already in Unity Catalog. Complete the Lakebase Change Data Feed (Autoscaling) template first so lb_<entity>_history tables exist in the bronze schema. This template reads from those tables; it does not create them.