Data Engineering Weekly #284

以下正文同步自 Data Engineering Weekly，版权归原站所有，已转换为易读排版。

How to Build a Data Platform From Scratch

We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single platform.

In this book, you’ll learn:

- How composable architectures allow teams to ship faster

- Why data quality matters and how you can catch issues before they reach users

- What observability means and how it will help you solve problems more quickly

Download your copy now

Editor’s Note: Leetdata.ai launching Cohorts & Ontology Podcast

Most bootcamps focus on theory and a few sample exercises, but nothing compares to learning from real-world implementations. We mine all the learning shared by companies in Data Engineering Weekly and build a cohort for learners. You can learn now on leetdata.ai- not only sql problems, but systems, modeling, and interviews- and you can share your experience too.

As you know, we are restarting the podcast, initially focusing on taxonomy, semantic modeling, ontology, and knowledge graphs. If you have domain expertise in these areas, please reach out to us. We would love to host a conversation with you.

https://leetdata.ai/cohorts

DuckDB: A Preview of DuckDB v2.0

An embedded analytics engine becoming network-addressable is a sharper change than another batch of SQL syntax. The DuckDB 2.0 feature preview presents a broader operating model: DuckDB servers via Quack and CONNECT asynchronous reads from object storage, and an extension API for longer-lived deployments. In-process use remains central, while these additions make it easier to place lightweight analytical services closer to the data.

https://duckdb.org/2026/08/17/duckdb-20-highlights

StreamFusion: Streaming for the AI age

Columnar execution has already reshaped batch engines, and streaming is beginning to follow the same path. This article shows how Flink’s planning, checkpointing, and SQL model can remain in place while native Arrow and DataFusion operators process supported plans. Row-oriented Kafka pipelines stay important, while columnar sources and sinks create a clear opportunity for this execution model.

https://streamfusion.tech/blogs/streaming-for-the-ai-age/#streaming-for-the-ai-age

Joshua Yu: The Evolution of Ontology: From Formal Semantics to Knowledge Graphs, Decisions and AI Agents

Knowledge graphs have long helped teams organize entities and relationships; agent workflows are broadening the role of semantics. The author traces a path from formal ontology to graphs, decisions, and agents that use capabilities, policies, and effects. Business vocabularies remain the foundation, while operational semantics give agents a clearer framework for acting responsibly.

https://medium.com/@yu-joshua/the-evolution-of-ontology-from-formal-semantics-to-knowledge-graphs-decisions-and-ai-agents-dfe8814dcc1b

Sponsored: AI Modernization Guide

AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.

Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.

Download the free guide

Netflix: Behind the Scenes: Evolving Netflix’s Ads Event Pipeline for Live — Part II

Streaming teams have spent years bringing enrichment closer to the event path. Netflix describes extending that pattern to live advertising using a stateful Flink join, regional routing, and a Spark recovery flow for late events. Central registries continue to serve governance needs, while the stream becomes the place where time-sensitive correlation can happen at scale.

https://netflixtechblog.medium.com/behind-the-scenes-evolving-netflixs-ads-event-pipeline-for-live-part-ii-826ebf9ad9fb

Leandro Vaz: Benchmarks against Gluten and Comet

Vectorized execution has already given Spark teams several ways to accelerate SQL workloads. The author adds FuseCore to a comparison with Gluten and DataFusion Comet, making the different execution approaches easier to study side by side. Benchmarks are most useful when they mirror the workload characteristics, and this comparison provides practitioners with another reference point for planning a pilot.

https://medium.com/@leandro-xonai/xonai-fusecore-benchmarks-against-gluten-and-datafusion-comet-2a9a5ea3a3f1

Booking.com: How we selected the next vector database at Booking.com

If you’re looking to adopt a vector database in your ecosystem, this is a good starting point as an evaluation matrix. Booking.com describes an evaluation with around 100 million embeddings, comparing performance alongside feature and operational requirements before choosing the next platform. Public benchmarks remain helpful, while production-shaped workloads give teams the richer context needed for a long-lived decision.

https://booking.ai/how-we-selected-the-next-vector-database-at-booking-com-1e738a5e3bb0

Netflix: A Tale of Two Flink Autoscalers

Autoscaling has become a core part of making streaming platforms approachable for more teams. Netflix compares a cluster-level approach with one based on per-operator true processing rates, then shows how workflow orchestration and graph-aware checks support stateful jobs. General defaults help platform adoption, while workload-specific controls make the resulting system more dependable.

https://netflixtechblog.com/a-tale-of-two-flink-autoscalers-e9f6a1b1492b

Guldmann: Data Architecture Patterns: Decisions for the AI Era

Data architecture has accumulated useful patterns across processing, storage, ownership, and transformation. The author takes an architectural view, places Lambda, Kappa, Medallion, Mesh, Lakehouse, and semantic architecture on distinct decision axes, then connects them through contracts. Each pattern retains its role, while shared meaning and governance provide a practical way for them to work together for AI.

https://guldmann.blog/2026/08/19/data-architecture-patterns-decisions-for-the-ai-era/

Chroma: wal3 - A Write-Ahead Log for Chroma, Built on Object Storage

This is a repeat read for me this week, and I did some back-of-the-envelope calculations to estimate the total cost of S3 PUT requests alone.

(26,280,000 / 1,000) × $0.005 = $131.40 per month

Not bad at all. In this case, we need to double it to account for the snapshot commit; nonetheless, given that a PUT can hold up to 64MB and a 100 ms flush rate is an interesting system design to read.

https://www.trychroma.com/engineering/wal3

Jimdoverse: How We Cut Our CDC Bill from $3,000 to $150 a Month

CDC has become a foundational pattern for moving operational changes into analytical systems. Jimdoverse reports a monthly cost reduction from $3,000 to $150 and uses it to examine the economics of a replication pipeline. Managed services remain valuable for many workloads, while matching capacity and pricing to actual change volume can create a leaner foundation.

https://www.jimdoverse.com/how-we-cut-our-cdc-bill-from-3-000-to-150-a-month-270f95495799

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.

正文由 FLUX 从来源站点 RSS 同步，内容未经改写；遇到排版缺失或需要图片、视频时请以原文为准。