How We’re Building Scam Alert on WhatsApp With End-to-End Encryption and Verifiability Guarantees

以下正文同步自 Meta Engineering，版权归原站所有，已转换为易读排版。

WhatsApp is committed to helping people stay safe while protecting the privacy of their messages. As scam tactics evolve — from impersonation to social engineering to AI-generated lures — we’re always evolving as well, so that our protections stay ahead of scammers while protecting people’s personal messages with end-to-end encryption.

Today, we’re sharing an early look at Scam Alert, a new, optional feature that runs an on-device machine learning model to alert a user about potential scam messages. No message content leaves the device for classification or is auto-reported to WhatsApp, Meta, or anyone else. The feature complements end-to-end encryption while enabling a user-controlled, optional scam alert when the model believes there’s a likely scam.

Before we make this feature available to all WhatsApp users, we are publishing this early technical overview alongside the feature’s limited rollout in Beta, and will continue working with our Bug Bounty community to stress-test this system. To help validate our implementation, we welcome feedback from the broader security research community.

Design Principles

Recent advances in on-device machine learning models make it possible to run accurate text classification entirely on mobile hardware without the performance, battery, or model-size tradeoffs that previously made on-device classification less practical. Scam Alert is well-suited to this approach: The model is small enough to run on-device, simple enough to publish for independent review, and effective without server-side components. The architecture we chose reflects a set of deliberate choices about what this system can and cannot do.

To that end, we designed Scam Alert to meet the following principles that adhere to the core guarantees of end-to-end encryption:

- On-device only: The model and the message data it processes all remain on the device.

- No automatic reporting: WhatsApp is unable to initiate sharing of any user data without the user’s action. The only way message content, or even the fact a scam was detected, reaches our servers is if the user explicitly chooses to report it, which is consistent with how user reporting works on WhatsApp.

- User control: Scam Alert is a user-controlled tool that provides additional information to the user, and the user can turn it off or on at any time.

How Scam Alert Works

Scam Alert is optional. Once the user turns it on, Scam Alert downloads a machine learning model to the device, where it runs inferences to classify whether incoming messages from non-contacts match known scam patterns. The model is trained on patterns observed in scam conversations from reports that users have sent to us. It performs probabilistic classification based on conversational structure and linguistic signals. No content is automatically reported to WhatsApp, Meta, or any third party.

If the model identifies a message as a likely scam attempt, the user sees a warning in the chat, which is not visible to the other person. From there, the user can decide what to do: block, report, or continue the conversation. If they decide that a warning is incorrectly flagged, the user can mark the chat as trusted, in which case the warning is removed and Scam Alert will not flag that chat again. If a user marks that they trust a chat, they can also opt in to share the last 5 messages received with WhatsApp to help improve the feature’s accuracy.

Foundational Safeguards

To uphold the principles above, we designed Scam Alert with the following foundational requirements and safeguards, with each architecturally enforced and independently verifiable by security researchers through an expanded bug bounty program and by users themselves through in-app logs.

- On-Device Processing and Privacy-Preserving Analytics: All inference happens on-device and no message content leaves the user device for classification. The minimal telemetry needed to measure whether the feature is working (i.e., aggregate and anonymous warning counts and user action counts) is processed within a confidential computing environment and sent to WhatsApp as differentially private aggregates. The confidential federated analytics pipeline is built on top of confidential virtual machines (CVMs), a type of Trusted Execution Environment (TEE). We chose this approach so its behavior can be independently verified.

- No Targeted Model Delivery: Neither Meta nor WhatsApp can deliver a specific model to a specific user. Every model version, including experimental variants, is published on a public transparency ledger before it is deployed.

- Verifiable Model Behavior: We publish model weights so that independent security researchers can verify that we’ve purpose built this for scams only.

The rest of this post details the technical implementation of each requirement.

On-Device Processing and Privacy-Preserving Analytics

As referenced above, all inference happens on-device. But we need to know that the feature itself is working – i.e., it is indeed catching real scams – and know if we need to update it to stay ahead of constantly evolving scams and improve the model over time.

To that end, our approach follows a set of data minimization principles. Message content does not leave the device, and logging is limited by design to only the signals that are needed to measure whether the feature is working as intended. Even those signals are processed within a confidential computing environment built on TEEs, which ensures that processing occurs in a secure environment that no one, including Meta and WhatsApp, can access. Our experience building and securing systems like Private Processing has informed the design of this system. Only anonymous, differentially private aggregates are made available to Meta and WhatsApp. Differential privacy works by adding carefully calibrated noise to provide a mathematical guarantee that adding or removing any single person’s data has a negligible effect on the anonymous, aggregated numbers. Hence these aggregates show how the feature performs across the population while telling us nothing about any individual.

For Scam Alert, that data is limited to two categories of approximate, aggregate counts:

- Warning Counts – how many times the on-device model surfaced a scam warning. This tells us whether the model is triggering at the right rate, which is essential for measuring precision and catching regressions across model versions.

- User Action Counts – when a user sees a warning, they can trust the sender or block and report. We log which action category was taken as an aggregate count. This tells us whether users find the warnings accurate, which is essential for measuring false positive rate.

Confidential Federated Analytics

To anonymize these warning and user action counts, we built a confidential federated analytics pipeline designed around the following privacy and security guarantees, each architecturally enforced and externally verifiable:

- On-Device Data Minimization: For Scam Alert, raw signals do not leave the device. The client aggregates them locally into counts and sends only those aggregated counts. These metrics are sent at randomized times, contain no device identifiers and limit any timestamp information to coarse time intervals. This ensures that neither the act of transmitting these metrics nor the metrics themselves can be used to identify a user.

- Confidential Processing: These metrics are processed within TEEs, secure hardware environments built on CPU-based confidential virtualization technologies, which allow attestation of software based in a hardware root of trust. Before any data is transmitted, the client checks these attestations and confirms them against a third-party log of acceptable binaries. Data is encrypted between the client and the TEE, so that no one in between, including Meta, WhatsApp, or any third-party relay, can access it.

- Secure Aggregation: Individual device metrics are not readable by Meta, WhatsApp, or anyone outside the TEE. They are merged into running aggregates, and only aggregated statistics, above a minimum cohort size and with differential privacy noise applied, are made available to Meta or WhatsApp. For Scam Alert, the device sends pre-aggregated counts directly to TEE for secure aggregation.

- Enforceable Guarantees: Before any data is transmitted, the client verifies that the code running in the TEE matches what was published on the third-party ledger and that the privacy parameters (such as differential privacy ε and δ, and k-anonymity thresholds) meet locally enforced guardrails. If verification fails or the privacy parameters are insufficient, the client refuses to transmit data. Any attempt to modify the processing guarantees either causes the system to fail closed or is publicly discoverable.

- Encrypted Recovery Checkpoints: Because the pipeline aggregates data over long periods, the system periodically saves encrypted checkpoints of its in-progress aggregates, so that a crash does not force a measurement to restart from scratch. These checkpoints contain only the partial aggregate counts already being computed, and they are encrypted: the keys never leave the TEEs, so only confidential federated analytics TEEs running the same attested binary can decrypt a checkpoint, and neither Meta nor WhatsApp can read it. Checkpoints are retained only for the bounded period needed to recover.

- Non-targetability: An attacker cannot target a particular user without attempting to compromise the entire system. All metrics are routed through an OHTTP relay that strips the requester’s IP address, and authenticated using anonymous credentials so that the system can verify that metrics come from a legitimate WhatsApp client without knowing which one. This limits the impact of small-scale attacks by ensuring that they cannot be used to target the data of a specific user.

- Verifiable Transparency: we will provide in-app capabilities for users to review what data was shared with the confidential federated analytics pipeline, the privacy parameters applied (such as differential privacy ε and δ, and k-anonymity thresholds), and details of how each secure session was established. We will be publishing the CVM image binary powering the pipeline, along with the source code of its privacy-relevant components, so that security researchers can independently verify that the published code is exactly what runs in the TEE. We will be expanding our Bug Bounty program to include the confidential federated analytics pipeline and will publish a detailed engineering white paper on its design.

The foundations of this pipeline were established in Meta’s peer-reviewed federated analytics work, publicly outlined in “PAPAYA Federated Analytics Stack: Engineering Privacy, Scalability and Practicality” (USENIX NSDI 2025). Here, we describe how Scam Alert applies and extends that foundation.

How Confidential Federated Analytics Works

The pipeline works as follows:

- On-Device Data Collection: Data is collected and stored in a dedicated local store, isolated from other application data. The confidential federated analytics system can only access data that the application has explicitly made available to it. Hardcoded privacy guardrails enforce data lifetime, scope, and access. For Scam Alert, this is only counts of warning events and user actions.

- Job Selection: At randomized intervals, when the device is idle and subject to a self-enforced daily resource limit, the client connects to the application server via OHTTP. The client authenticates using anonymous credentials that prove it is a legitimate WhatsApp client without revealing which one, and fetches the list of active jobs for the pipeline. For each job, the client checks whether the privacy parameters (ε, δ, and k-anonymity thresholds) meet locally enforced guardrails, whether the device has new metrics to send, and whether participating would exceed its daily limits. The client can reject any

正文由 FLUX 从来源站点 RSS 同步，内容未经改写；遇到排版缺失或需要图片、视频时请以原文为准。