ZGateway: Learnings from Putting a Proxy in Front of ZippyDB

以下正文同步自 Meta Engineering，版权归原站所有，已转换为易读排版。

- We’re introducing ZGateway, the proxy we are using to unify traffic through ZippyDB, Meta’s most widely-used key value store.

- As a bonus, it also enables admission control, load balancing, cross-region resilience, and richer operations.

ZippyDB is the most widely used key value store at Meta, backing product metadata, counters, and configuration, and can serve billions of operations per second across a globally distributed fleet.

A previous post described how ZippDB works. This post is about the layer in front of it: ZGateway, the proxy through which we are unifying ZippyDB client traffic.

ZGateway grew out of the need to manage the sprawl of ZippyDB’s client fleet, and its value turned out to be structural. A ZippyDB client might be one of more than a million hosts, owned by hundreds of teams, that we cannot change quickly. A proxy sits in a different position entirely — in the path of many clients at once — and that shared vantage point lets it do what no single client could.

Every capability below is easier, safer, or only possible at all in one managed tier rather than in a million client binaries. We tell the story through the two clearest cases, connection management and request batching.

Why ZippyDB Needed a Proxy Layer

Proxies show up wherever a large, diverse client population talks to a shared backend: a connection pooler, a service mesh sidecar, a CDN edge, an API gateway. Interposing between many callers and a shared resource buys three things.

- It bounds the problem: The backend stops seeing the client population and starts seeing a fleet its own operators control

- It creates a home for shared work — pooling, retries, routing, caching, admission control — solved once by the team that knows the backend best, instead of in every client.

- It creates a control point: The only place you can see the whole workload, attribute load to whoever generated it, and change behavior in minutes rather than waiting out a fleet-wide client rollout.

The tradeoff is a hop and one more tier to operate; the trade pays off when the client population is large, diverse, and not yours to change. ZippyDB is an extreme case.

In the direct-access model, every ZippyDB client connects to every database host it needs. A single client can touch tens of thousands of distinct shards in a stable window; those shards sit across hundreds of thousands of database hosts. The result is a dense many-to-many mesh of TLS connections: A typical client holds tens of thousands of outbound connections, and a typical database host can accept tens of thousands of inbound ones.

Figure 1: Direct access produces an unbounded many-to-many mesh; ZGateway collapses client fan-out and database-host fan-in into bounded numbers.

That mesh is wasteful and fragile. Every open connection consumes memory, CPU, and a file descriptor on both ends, mostly while idle, and inbound connections grow with the client population, so every new client cohort makes every database host worse. And because each client manages its own pooling and failover, a sudden drop in connection reuse — a cohort restarts, a deploy rolls — hits the fleet with a storm of new connections; we have traced host crashes from file-descriptor exhaustion and OOMs to exactly this.

It is hard to fix client-side because two systems move at once. Client fleets keep changing pooling policies and expanding while the database fleet consolidates on its own schedule. Coupling them directly feels like jumping between two moving cars. A proxy decouples them, collapsing the mesh into two bounded hops — a win on efficiency, performance, scalability, and above all reliability.

That last one matters most. Under direct access a reconnection storm becomes a catastrophe. In one incident a routing bug led every client to open a connection per shard, hosts breached their file-descriptor limit, and the fleet fell into a reboot loop. With ZGateway in the path that storm is contained at the proxy tier — a fleet we control, observe, and can harden centrally. Connection management does not disappear behind a proxy; it moves to the one place we can solve it.

Direct access was the right design for the initial part of ZippyDB’s life. Fan-in scales with the client population, so the resource use grows with adoption: at a few thousand clients the mesh is an inefficiency, at scale it is a reliability limit. The means arrived on the same schedule — ServiceRouter feature improvements, Thrift overload protection, a thin client, and other features are what make a shared tier feasible, and building ZGateway earlier would have meant building each of those first.

What ZGateway Is

ZGateway is a stateless proxy tier between ZippyDB clients and the database (ZServer) fleet. It can handle greater than 1 billion operations per second and carries about 40% of all ZippyDB traffic, projected to grow past 60%, while adding only about 6% computational overhead to an average use case. It runs as regional tiers discovered through ServiceRouter, Meta’s hyperscale service mesh solution, keeping every client near its gateway, in two flavors sharing one pipeline: a pure proxy and a read-through cache. it runs our thick C++ client as its engine, one internal client per use case. ZGateway is a ZippyDB client run as a managed service, which made moving capability here natural.

Figure 2: The ZGateway request path, from a client’s sticky regional connection to the ZServer replicas.

A client sends a request over its sticky connection to a regional ZGateway host, which terminates TLS, authorizes it against the use case’s ACLs, and applies per-tenant admission control, validation, and shaping. ZGateway resolves the shard, checks the local cache on a caching tier, and batches and coalesces misses and writes with other in-flight requests for that shard before sending them to the correct replicas. Responses are demultiplexed back to the callers, with per-use-case metrics, traces, and quota usage recorded along the way.

The key property is the asymmetry of connection counts. Each client needs only a sticky pool to its regional ZGateway hosts, and each ZServer sees connections only from the ZGateway fleet, whose size we control. Some responsibilities deliberately stay put — TLS in the Thrift/ServiceRouter stack, key-to-shard mapping in the shard locator, replica selection and hedging in the embedded client. ZGateway owns traffic management, not a reimplementation of the database client.

The Fan-In/Fan-Out Reduction

Here is the arithmetic:

Model the fleet as balls into bins. Throw B balls (shards a host touches) into H bins (hosts) and count distinct bins hit.

The expected number is:

And a specific bin is hit with probability:

Fan-out is distinct bins hit; fan-in is that probability times the caller population. With round mock figures — 20 regions, 500,000 database hosts, 30,000 proxy hosts, 1,000,000 clients, 50,000 shards per client — the model yields:

Figure 3: Per-host connection counts collapse by ~97–98%. (Model-based estimate; the per-pair TLS multiplier cancels in the ratio).

The connections don’t vanish; they move to the tier built to hold them. End to end, total persistent connections still drop by roughly 19x, because each backend connection multiplexes many clients.

But the one-time reduction, however dramatic, isn’t the real point. The real point is the change in scaling behavior. In the direct model, database-host fan-in is H_client · p — linear in the client population, so every new cohort makes every database host worse. With ZGateway the client population drops out entirely: fan-in reduces to approximately R · S_host, regions times shard density per host, independent of both fleets. The only lever left is shard density, which we own. An unbounded number driven by everyone else becomes a bounded number we control.

Collapsing the Request Stream: Batching and Coalescing

Because ZGateway sits in the path of many clients it can do what no client library can – combine work across unrelated callers. A shared batcher on each host groups requests headed for the same destination, keyed by use case and physical shard, and merges them into one backend RPC.

It also coalesces. If several callers want the same key at the same moment, the gateway fetches it once and fans the result out. A client-side batcher can only merge its own process’s requests; the gateway collapses them across clients.

Every RPC carries fixed overhead regardless of size — Thrift serialization, shard lookup, authorization, syscalls — so folding many operations into one amortizes all of it, meaning fewer, larger backend requests, lower QPS and CPU, and steadier load as the linger window smooths micro-bursts. And because a use case is billed by the QPS it sends, batching stretches its rate-limit budget, cutting throttling with no work on its side.

Two of batching’s benefits aren’t about efficiency at all. The first is what coalescing does under a hot key. Thousands of simultaneous callers collapse into a single backend read, so a hot key can never become a stampede against one replica.

The second is what it lets us remove. For years, customers who wanted batching ran client-side libraries that were fragile, CPU-hungry, individually tuned, and a steady source of incidents because that complexity lived in a million binaries we didn’t control. A shared batcher batches across clients, which those libraries cannot — and let us retire them.

Each request is parked in an in-memory batch that flushes when a linger window elapses, the payload crosses a size limit, or the request count hits a cap, so added latency stays bounded; oversized or just-migrated batches fall back to individual sends. Holding requests in memory is an OOM risk, so batching ships with two safety mechanisms.

Idle eviction handles slow-burn growth: batch-map entries idle beyond a TTL are erased on the next flush. An in-flight cap handles acute overload. When the backend slows, coroutines running flushed batches pile up faster than they drain, so the cap rejects new executions once the count crosses its limit. Steady hygiene plus an acute safety valve is what makes batching safe by default.

Figure 4: Batching and coalescing across clients.

How ZGateway Evolved

Once traffic flows through one tier, it becomes the natural home for capabilities every client would otherwise reimplement. Batching is the clearest example; here are the rest.

Traffic Routing and a Safe Migration Path

Moving traffic onto a proxy is a high-stakes migration. It must be incremental, reversible, and scoped. Routing to ZGateway is controlled by client-side configuration flags scoped per service and shard prefix: a percentage knob ramps eligible traffic, a region filter limits blast radius, a global kill switch gives instant rollback. It is pure configuration, with no client code change, which makes the rollout controllable in real time.

Tenant Isolation and Admission Control

A shared tier serves hundreds of use cases, so one misbehaving tenant must not starve the others. ZGateway’s defense is Discriminant Load Shedding (DLS). Every request maps to a per-tenant bucket, keyed by use case and split by priority, and buckets drain round-robin. When a tenant floods the tier, its own bucket fills and its excess is shed while every other bucket keeps draining — isolation as a property of the structure, not of luck. In front of DLS, a CPU concurrency controller uses an AIMD loop to adjust how fast the shared token bucket admits work; a memory handler guards against OOM the same way.

The shedding stays discriminant. In a controlled overload at >90% CPU across roughly 1,350 active tenant buckets, only 6 — the actual noisy neighbors — were shedding; the other ~1,344 executed 99.9% of their requests with zero rejections, goodput held near 97–98%, and the machinery cost about 8% of CPU.

Figure 5: Discriminant load shedding under overload.

Read Caching With Live Invalidation

On a cache tier, hot reads are served from an in-process

正文由 FLUX 从来源站点 RSS 同步，内容未经改写；遇到排版缺失或需要图片、视频时请以原文为准。