Data Engineering Weekly #285

以下正文同步自 Data Engineering Weekly，版权归原站所有，已转换为易读排版。

How to Build a Data Platform From Scratch

We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single platform.

In this book, you’ll learn:

- How composable architectures allow teams to ship faster

- Why data quality matters and how you can catch issues before they reach users

- What observability means and how it will help you solve problems more quickly

Download your copy now

Jacob Peake: AI Chip Architectures

We often feel the hardware design is an order of magnitude removed from our data work. The principles of mechanical sympathy taught us otherwise. The article highlights that each major player makes fundamentally different architectural bets to solve this data-movement problem.

https://www.jepeake.com/ai-chip-architectures

Pramod Sadalage & Prem Chandrasekaran: Making Your Data Ready for Agentic AI

To make data "AI-ready," the authors argue you must build a foundational stack composed of four key pillars:

- Trusted data via data contracts

- Better traceability & governance

- The context layer

- The governance

I think, as an industry, we haven’t settled on whether we need data contracts, but the authors highlight that this is the table stakes.

https://martinfowler.com/articles/making-data-ready-for-agentic-ai.html

Ian Macomber: The Shape and Feel of the Post-AI Data Stack

The author reemphasizes and sketches out the thoughts from the previous blog as a practical implementation for making data AI-ready in the post-AI era. In late Feb, I called out ECL (Extract, Contextualize, and Link) as the emerging pattern in the blog "Data Engineering after AI". I’m glad to see these patterns coming into reality.

https://www.iandmacomber.com/blog/post-ai-data-stack/

Sponsored: AI Modernization Guide

AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.

Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.

Download the free guide

Grab: Data Mesh at Grab (Part III): Operationalizing data reliability with automated DPIs

Grab writes about building an automated workflow to detect and manage data contract breach issues. When a breach occurs, the system triages the alert and utilizes specialized diagnosis architectures to identify the root cause and automatically execute recovery strategies for routine failures.

https://engineering.grab.com/data-mesh-at-grab-part-three

Airbnb: Project Lighthouse — Part 3: Introducing project-lighthouse-anonymize

Airbnb has open-sourced the Python library that powers Project Lighthouse, an initiative designed to measure disparities in user experiences while strictly preserving data privacy. The code release is accompanied by two new technical papers that detail the scalable k-anonymity algorithm used and introduce a comprehensive framework for assessing the quality of anonymized data.

Github: https://github.com/airbnb/project-lighthouse-anonymize/

https://medium.com/airbnb-engineering/project-lighthouse-part-3-introducing-project-lighthouse-anonymize-74f8b26653fb

Booking.com: Beyond the Dashboard: Accelerating Real-Time Intelligence in the Age of AI

Booking.com highlights a bottleneck in traditional data reporting, where reliance on engineering teams for ad hoc SQL queries and dashboards delays critical business decisions. To solve this, it proposes a "Talk to your data" ecosystem that pairs GenAI—specifically Snowflake Agents—with a structured semantic layer, enabling non-technical users to query data in natural language securely.

https://medium.com/booking-com-development/beyond-the-dashboard-accelerating-real-time-intelligence-in-the-age-of-ai-6f1f0f9c123f

Mimoune Djouallah: Writing Parquet That VertiPaq Likes

The author explores how to manually optimize Parquet file layouts to improve read performance and reduce compute capacity consumption for Power BI's VertiPaq engine. The finding is interesting: VertiPaq performs best when Parquet files feature fully dictionary-encoded columns, row groups sized between 2 and 6 million rows, and a global sort that prioritizes low-cardinality columns. TIL about Fabric’s V-Order optimization.

https://datamonkeysite.com/2026/08/29/writing-parquet-that-vertipaq-likes/

Chris Douglas: Compaction Maps

Table-format compaction optimizes data layouts for efficiency but frequently causes conflicts that force concurrent transactions to re-execute from scratch. To address this, the author introduces "compaction maps," which are compact data structures that quickly resolve these conflicts by mapping row-deletion references from the old layout to the new one. The hope here is that, by avoiding full re-execution, the solution may drastically reduce layout conflict resolution times from minutes down to mere milliseconds.

https://cdouglas.github.io/posts/2026/08/compaction-maps

Vignesh Ravichandran: The small-file problem gets worse with CDC

CDC-type workloads are where Iceberg performance goes wild. The small-file problem: a global lock in compaction and merge is where you start to feel the design issues in Iceberg. The experiment is a bit of a surprise; can a change in where metadata lives make an impact? The author says that by the benchmark, DuckLake's performance is way better than Iceberg's on CDC workloads.

https://streambed.dev/blog/ducklake-target-support/

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.

正文由 FLUX 从来源站点 RSS 同步，内容未经改写；遇到排版缺失或需要图片、视频时请以原文为准。