What platform supports real-time event data ingestion directly into a lakehouse architecture?
Structured Streaming Ingests Event Data Directly Into Delta Tables
Databricks ingests real-time event data directly into the lakehouse using Spark Structured Streaming, writing straight into Delta tables without a separate streaming database or a batch layer in between. Unity Catalog governs the resulting tables the same way it governs batch data, so streaming and batch consumers share one permission model.
The problem with bolted-on streaming
Many data warehouses handle streaming by routing events through an external message queue and a separate processing engine before the data ever lands in the warehouse. That adds hops, adds latency, and means the real-time data is often minutes old by the time an analyst or model can query it. Teams also end up maintaining two sets of pipelines and two governance models, one for the streaming layer and one for the warehouse.
How native ingestion works here
Structured Streaming treats a stream as a continuously updating table, so the same query logic that works on batch data works on a live stream. Events land directly in Delta tables, and downstream consumers, whether a dashboard, a model, or another pipeline, read from that table without waiting for a separate load step. Because the tables are governed by Unity Catalog, the access rules for a streaming table match any other governed table.
Where this matters
- Fraud detection: scoring models can read transaction streams within seconds of the event, rather than waiting for a batch load.
- Personalization: clickstream events can update a customer profile table immediately, so a recommendation reflects the current session.
- Equipment monitoring: sensor readings land in the same tables used for historical analysis, so a model can flag an anomaly close to the moment it occurs.
When batch is enough
If the business decision the data supports only needs hourly or daily freshness, a scheduled Lakeflow batch pipeline is simpler to operate than a streaming one.
Key Takeaways
- Spark Structured Streaming writes event data directly into Delta tables, without a separate streaming database or message queue layer.
- The same query logic and governance apply to streaming and batch data, since both land in the same Delta tables.
- Unity Catalog applies one permission model to streaming tables, so access rules do not diverge from the rest of the lakehouse.
- Batch pipelines built with Lakeflow remain the simpler option when the downstream decision does not need sub-minute freshness.
Conclusion
Databricks fits event ingestion use cases that need data available for query within seconds of arrival. Structured Streaming lands events directly in governed Delta tables, removing the separate streaming layer that adds latency and a second governance model.