Data Engineering Weekly #287

以下正文同步自 Data Engineering Weekly，版权归原站所有，已转换为易读排版。

How to Build a Data Platform From Scratch

We wrote an eBook on Data Platform Fundamentals to help you build a data platform like the happy teams running on a single platform.

In this book, you’ll learn:

- How composable architectures allow teams to ship faster

- Why data quality matters and how you can catch issues before they reach users

- What observability means and how it will help you solve problems more quickly

Download your copy now.

Editor’s Note: Upcoming Tech Talks

I’m excited to step out from behind the architecture docs and code to share what I’ve been building over the past three months. If you’re local to the Bay Area or attending the fall streaming circuit, I’ll be speaking at two upcoming events:

- Data for AI: Access to Understanding Meetup (September 16th · Menlo Park)

Topic: The Knowledge Spine: A Machine-Executable Ontology for Governed Marketing Activation

As agentic systems and contextual AI mature, the bottleneck has shifted from model capability to semantic governance. I’ll be breaking down how to build an active, machine-executable ontology layer that translates raw audience data into deterministic, governed action.

https://luma.com/dataforai-h5ua

- Data Streaming Summit 2026 (October 8th - San Francisco)

Topic: One Partition, 2,000 Tenants: Order-Preserving Fair Kafka Consumption with an LSM in Object Storage

Multi-tenancy in event streaming rarely plays nicely with strict ordering and noisy neighbors. I’ll dive into the mechanics of pairing Kafka with an LSM-tree abstraction backed by object storage to achieve fair, per-tenant ordering at scale without partition explosion.

https://datastreaming-summit.org/dss2026

If you’re planning to attend either event, please find me and say hi—I’d love to catch up, hear what you’re working on, and talk shop.

OpenAI: Rapidly scaling online storage to serve over 1 billion ChatGPT users

OpenAI writes about how it evolved its online storage platform, Habitat, from a client-side Python library into a centralized distributed service managing over 500 petabytes of data and 70 million requests per second. The pattern seems an interesting development methodology in the age of AI: write 0-1 to optimize for developer productivity, and then rewrite to scale.

https://openai.com/index/scaling-storage-one-billion-users-part-one/

Pinterest: Evolving Pinterest’s Embedding Retrieval Platform

Embeddings are rapidly becoming a core data engineering workload because transforming massive amounts of unstructured data into vector representations requires building robust, scalable pipelines to continuously generate, index, and serve these high-dimensional assets efficiently. This blog post details how Pinterest is evolving its in-house search platform, Manas, to manage tens of billions of embeddings while curbing infrastructure costs.

https://medium.com/pinterest-engineering/evolving-pinterests-embedding-retrieval-platform-aede4e831e01

Sponsored: Consolidate your fund admin data

Trade the quarterly fund admin data fire drill for one consolidated view of every source your firm runs on with Aquata®. See what’s possible with:

- Pre-built connectors

- Consistent data models oversight

- Normalized performance reporting

Book a Demo

BlaBlaCar: (Re)Building an AI-Ready data universe at BlaBlaCar

Continue on the changing landscape of data engineering post-AI: along with embeddings, extracting undocumented business logic from unstructured data, notes, or other forms of data is becoming essential. BlaBlaCar writes about its legacy data infrastructure, complex technical debt, undocumented business logic, and unoptimized queries, leading to high maintenance costs and unreliable AI outcomes. The blog narrates how it completely rebuilt its data warehouse into a modular, explicitly documented, and strictly governed AI-ready architecture using dbt and an automated dual-agent AI workflow.

https://medium.com/blablacar/building-an-ai-ready-data-universe-at-blablacar-cb15fbc42020

Thumbtack: Three principles for building a vector platform at Thumbtack

Thumbtack made vector search boring in the best way: reuse existing Postgres, pipelines, and access patterns instead of building a bespoke AI stack.

Continuing the theme that embeddings are becoming a critical component of data engineering, Thumbtack highlights that the key design choice is treating embeddings as governed, durable data—not an application-side afterthought.

https://medium.com/thumbtack-engineering/three-principles-for-building-a-vector-platform-at-thumbtack-bca5a33dca16

Sponsored: AI Modernization Guide

AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.

Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.

Download the free guide

Glance: Building Resumable Real-Time LLM Streaming with Redis Streams

Glance writes about preventing AI responses from disappearing when a user refreshes a page or loses connectivity. It saves each piece of the response in a Redis Stream as it is generated, so the app can replay the missing pieces after reconnecting. The design separates AI generation from the user’s browser connection, avoiding unnecessary re-runs.

https://engg.glance.com/building-resumable-real-time-llm-streaming-with-redis-streams-09cfa9e79358

Meta: Measuring What Matters: How Causal Inference Turned AI Infrastructure Into a Quantifiable Business Investment

Meta writes about using causal inference to measure whether changes to AI infrastructure actually improve business outcomes. Instead of trusting before-and-after metrics, the method estimates what would have happened without the change. This gives infrastructure teams a clearer way to prioritize investments based on measured impact, not correlation.

https://medium.com/@AnalyticsAtMeta/measuring-what-matters-how-causal-inference-turned-ai-infrastructure-into-a-quantifiable-business-bbc49bddaa2b

Remitly: When Metrics Backfire: The Cobra Effect in the Age of AI

Remitly writes about the Cobra Effect: when people or AI systems optimize a target number while making the real outcome worse. In the age of AI, models can learn to game the metric they are given, producing results that look successful on a dashboard but miss the actual goal. The article argues that teams should use several measures and watch for unintended behavior instead of treating one metric as the truth.

https://medium.com/remitly/when-metrics-backfire-the-cobra-effect-in-the-age-of-ai-3e14c50d9abe

Razorpay: Five Years of Kafka at Razorpay’s UPI Switch

Razorpay writes about using Kafka to carry payment events through its UPI Switch, where processing must remain reliable as the system grows. Over five years, the team evolved the platform so services can handle and recover payment events independently without losing the overall transaction flow. The story shows that, for payments, streaming infrastructure is as much about correctness and recovery as it is about speed.

https://engineering.razorpay.com/tryst-with-kafka-2f5cef766c45

Housing: We Cut Cloud Waste Before Touching Cluster Sizes: Lessons from Running a Data Platform

Housing.com writes about finding cloud waste in forgotten storage and retention settings before shrinking compute clusters. The team used Databricks System Tables to identify unused BigQuery data, tune retention rules, and right-size data-ingestion services. The lesson is that better visibility and guardrails can reduce spending without risking platform performance.

https://medium.com/engineering-housing/we-cut-cloud-waste-before-touching-cluster-sizes-lessons-from-running-a-data-platform-9ea96a1f9fbe

All rights reserved, Dewpeche Private Limited. I provide links for informational purposes and do not endorse them. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.

正文由 FLUX 从来源站点 RSS 同步，内容未经改写；遇到排版缺失或需要图片、视频时请以原文为准。