Data Engineering Weekly #288

以下正文同步自 Data Engineering Weekly，版权归原站所有，已转换为易读排版。

How to Build a Data Platform From Scratch

We wrote an eBook on Data Platform Fundamentals to help you build a data platform like the happy teams running on a single platform.

In this book, you’ll learn:

- How composable architectures allow teams to ship faster

- Why data quality matters and how you can catch issues before they reach users

- What observability means and how it will help you solve problems more quickly

Download your copy now.

Mark Rittman: dbt Summit 2026 - What’s New and What’s Coming from dbt Labs, Fivetran and the Agentic Data Stack

Great summary of the recent dbt summit, capturing the critical announcements and the future vision. dbt charts are something interesting to try out in an embedded & iterative pipeline build. It is a bit weird to see a managed data lake, Fivetran, and Open Data Infrastructure in one slide, though. 😊

https://blog.rittmananalytics.com/dbt-summit-2026-whats-new-and-what-s-coming-from-dbt-labs-fivetran-and-the-agentic-data-stack-52bbef9334a9

Tim Poterba: The DAVE stack: the age of domain-optimized analytical databases on Datafusion, Arrow, and Vortex, Embedded

Discard the DAVE one. The article touches on the question most enterprises should be asking right now. Do we really have a big data problem? If so, how do you define this big data? Do we need complex distributed computing machines to crunch the numbers? The author answers the same question with Phoebe: a domain data system for genetic analytics. For most enterprise systems, if you get the incremental data-processing design right, you don’t have a big data problem.

https://sequenceandsilicon.substack.com/p/the-dave-stack-the-age-of-domain

Joanna He: Beyond the Semantic Layer: Engineering the Agentic Data Stack

The author writes about preparing data systems for AI agents, which cannot reliably work from raw tables and undocumented business rules. The proposed stack keeps metrics, relationships, permissions, and business context as version-controlled files, then lets agents read that context through a controlled gateway before running queries. The aim is to give agents shared definitions and safety checks, reducing wrong answers and making data access easier to govern.

https://medium.com/@Joannahe/beyond-the-semantic-layer-engineering-the-agentic-data-stack-98fbaee9ad10

Sponsored: AI Modernization Guide

AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.

Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.

Download the free guide

Airbnb: The guest journey, updated in real time: extending Airbnb’s sequence recommender with Chronon

Airbnb writes about making its search recommendations reflect a guest’s latest browsing instead of waiting for a daily update. Its Chronon platform reacts when a guest views a listing or searches, combines that event with their existing history, and refreshes their profile with the same recommendation model. This lets the next search use more current intent while keeping the model off the fast search path.

https://medium.com/airbnb-engineering/the-guest-journey-updated-in-real-time-extending-airbnbs-sequence-recommender-with-chronon-8f1582578553

Canva: Worker Backpressure

Canva writes about preventing queue workers from overwhelming a dependency that is already failing. Each worker tracks successes and failures, lowers how many messages it processes at once when errors rise, and automatically increases speed as the dependency recovers. This keeps work safely in the queue instead of flooding it with failed messages and gives strained services room to recover.

https://www.canva.dev/blog/engineering/worker-backpressure-part-1-how-we-taught-our-queue-workers-to-slow-down/

Booking.com: How We Built Automated Capacity Testing for Kafka Consumers

Booking.com writes about replacing risky, manual Kafka failure drills with automated capacity tests. The system temporarily gives one consumer more Kafka partitions, checks its health after each step, and automatically restores normal assignments if the test ends or its controller fails. This gives teams repeatable evidence of how much extra work a consumer can handle when another instance goes down.

https://medium.com/booking-com-development/how-we-built-automated-capacity-testing-for-kafka-consumers-1853623bce78

Orb: Debugging large datasets with DuckDb

Orb writes about debugging billing-usage data that reaches its S3 intake but is rejected during validation or duplicate checks. The team uses DuckDB to query customer files in place, compare raw events with Orb metrics, and identify issues such as duplicate event keys without copying data or operating a separate analytics system. This gives support a fast way to find source-data problems before they become billing mismatches.

https://www.withorb.com/blog/debugging-large-datasets-with-duckdb

Ian Binder: Stop Paying NAT Gateway Prices for S3 Traffic

The author explains how to avoid unnecessary NAT Gateway charges when private AWS workloads send data to S3. A free S3 gateway endpoint gives each private subnet a direct AWS-internal route, but it must be associated with every relevant route table to take effect. This simple routing change keeps S3 traffic off the paid NAT path and can reduce network costs without changing the application.

https://medium.com/@ianbinder/stop-paying-nat-gateway-prices-for-s3-traffic-3944ccd64bf8

Bolt Labs: Migrating 10 million monthly queries: How Bolt moved Looker from Presto to Databricks

Bolt writes about moving its Looker reporting workloads from Presto to Databricks after peak demand exposed reliability, maintenance, and consistency problems. The team built a migration tool that understands both SQL and Looker’s Liquid templates, then compared dashboard results on both systems and kept every change reversible. The move consolidated reporting on one query engine, reducing operational complexity while protecting the numbers users rely on.

https://medium.com/bolt-labs/migrating-10-million-monthly-queries-how-bolt-moved-looker-from-presto-to-databricks-f427666a0de6

All rights reserved, Dewpeche Private Limited. I provide links for informational purposes and do not endorse them. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.

正文由 FLUX 从来源站点 RSS 同步，内容未经改写；遇到排版缺失或需要图片、视频时请以原文为准。