Data Engineering Weekly #286

以下正文同步自 Data Engineering Weekly，版权归原站所有，已转换为易读排版。

How to Build a Data Platform From Scratch

We wrote an eBook on Data Platform Fundamentals to help you be like the happy data teams operating under a single platform.

In this book, you’ll learn:

- How composable architectures allow teams to ship faster

- Why data quality matters and how you can catch issues before they reach users

- What observability means and how it will help you solve problems more quickly

Download your copy now

Tivadar Danka: The Roadmap of Mathematics for Machine Learning

AI is rapidly changing the landscape, making a strong foundation more necessary than ever before. The author highlights the roadmap for the maths foundation needed to prepare for machine learning.

https://thepalindrome.org/p/the-roadmap-of-mathematics-for-machine-learning

Instacart: Agentic Machine Learning Modeling at Instacart

Instacart writes about integrating AI agents into its Griffin ML platform to autonomously generate, code, and evaluate machine learning hypotheses under human supervision. Instead of basic hyperparameter tuning, agents dynamically adapt strategies, test diverse model families, and compound small architectural improvements based on real-time evaluation feedback.

https://tech.instacart.com/agentic-machine-learning-modeling-at-instacart-fb3ecd295ee7

Netflix: The Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scale

Netflix outlines a four-phase lifecycle (Birth, Training, Deployment, and Monitoring) to build and maintain a reliable, human-anchored LLM-as-a-Judge system for evaluating machine-generated text at scale. The authors introduce Reasoning-Aligned Rubric Tuning (RART) to optimize judge prompts by catching “right label, wrong reason” errors using written human rationales rather than relying solely on simple label agreement.

https://netflixtechblog.medium.com/the-lifecycle-of-llm-as-a-judge-building-aligning-and-monitoring-at-scale-c95bd8283508

Sponsored: AI Modernization Guide

AI is reshaping how data teams operate. But legacy pipelines, brittle workflows, and fragmented tooling weren’t designed for this shift.

Learn how leading teams are future-proofing their infrastructure before AI demands overwhelm it.

Download the free guide

Timothy Wong: The Semantic Layer and Data Modeling Dilemma in AI Analytics

The more time we invest in the semantic layer and data model, the more accurate and consistent the answers become. But it can also slow us down.

This is the real trade-off in any enterprise in preparing the data for AI. The author gives some practical suggestions from the start of core modeling.

https://www.linkedin.com/pulse/semantic-layer-data-modeling-dilemma-ai-analytics-timothy-wong-fwlfc/

StarTree: Scaling Apache Pinot Beyond Millions of Segments (or Parquet Files)

The control plane/metadata bloating is the surprising scalability challenge in operating data systems at scale. starTree introduces Segment Groups, an opt-in capability that aggregates metadata for ~100 underlying segments into a single control-plane representation without physically rewriting or merging the data. By changing the unit of coordination while preserving the fine-grained segment layout on data servers, Pinot clusters can scale to 10 million segments and efficiently process massive open-table datasets such as Apache Iceberg.

https://startree.ai/resources/scaling-apache-pinot-beyond-millions-of-segments/

Netflix: Running Apache Spark experiments in my sleep (and on a plane)

Netflix writes about its adoption of persistent tmux sessions to run parallel experiments to debug the Spark memory problems. Netflix streamlined this process with an AI agent that automated deployment, polling, and structured logging to enforce strict single-variable testing. The underlying memory bottlenecks within collect_list were ultimately fixed by unboxing primitives, stripping bloated string attributes, capping sessions before Window operations, and lowering AQE runtime broadcast thresholds.

https://netflixtechblog.medium.com/running-apache-spark-experiments-in-my-sleep-and-on-a-plane-588816b6b5f8

Lyft: Rerouting the Stream: How Lyft Moved to the Apache Flink Operator

Lyft writes about migrating its streaming fleet from a legacy in-house tool to the Apache Flink Kubernetes Operator to resolve mounting technical debt and access modern management features. The article details the zero-downtime transition by building a deploy API translation layer to dynamically map legacy custom resources and utilizing the upstream FlinkBlueGreenDeployments CRD for safe cutovers.

https://eng.lyft.com/rerouting-the-stream-how-lyft-moved-to-the-apache-flink-operator-36f20246d250

LinkedIn: Rebuilding LinkedIn’s Follows Recommendations with LLM-Based Semantic Retrieval and Ranking

LinkedIn writes about rebuilding its Follows Recommendation system by shifting from popularity heuristics to a unified bi-encoder LLM architecture that maps both members and creators into a shared semantic vector space using narrative prompting. The article discusses how it scales using Ray and FAISS for heavy offline candidate batch generation, alongside Proxima-hosted vector search for low-latency, real-time onboarding recommendations.

https://www.linkedin.com/blog/engineering/ai/rebuilding-linkedins-follows-recommendations-with-llm-based-semantic-retrieval-and-ranking

WMG: When the source of truth is a Google Sheet

This WMG Innovation Lab article describes how it built a resilient Databricks data pipeline to turn manually updated, messy Google Sheets used for RIAA counterfeit takedowns into a clean, queryable source of truth. It highlights key ingestion-layer strategies for handling schema drift, such as filtering out invalid monthly tabs, normalizing headers to snake_case, and using SHA-256 row fingerprints to distinguish edits from new inserts.

https://tech.wmg.com/handling-the-messy-reality-of-google-sheets-building-a-resilient-counterfeit-takedown-pipeline-d4c3502664bd

Wiz: 5 Billion Records, 10 Terabytes, Four Weeks: A Real Data Migration Playbook

Wix outlines practical strategies for tackling massive data migrations without stalling. It emphasizes uncovering legacy database corruptions early through dry runs, addressing parity gaps with end-to-end testing, and properly sizing Kafka partitions and database sharding to outrun incoming change rates.

https://www.wix.engineering/post/5-billion-records-10-terabytes-four-weeks-a-real-data-migration-playbook

All rights reserved, Dewpeche Private Limited. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.

正文由 FLUX 从来源站点 RSS 同步，内容未经改写；遇到排版缺失或需要图片、视频时请以原文为准。