An Organizational Second Brain: Building an AI That Learns From Experts

以下正文同步自 Meta Engineering，版权归原站所有，已转换为易读排版。

- We’ve built an AI agent that acts as a secondary expert for a given domain, making deep specialist knowledge readily available and preserved for anyone in an organization to access, share, and build upon.

- This is not a typical domain-specific agent. Its novelty comes from integrating two layers:

- A structured, auditable knowledge architecture separates what the agent knows from how it reasons.

- A self-improvement loop then compiles expert feedback into verified, regression-tested updates without model retraining.

- Together, these two layers turn one-off expert corrections into permanent, compounding institutional memory, and the pattern is designed to generalize to other domains governed by retrievable text rather than model weights.

- This system is saving domain subject matter experts (SME)s at Meta substantial time, allowing them to focus more on the work where their knowledge matters most.

Many large organizations have the same problem when it comes to specialist knowledge. While some of it is written down in the form of models, playbooks, checklists, and frameworks, the most valuable specialist knowledge lives in people’s heads and rarely gets captured anywhere durable. In compliance domains, for example, the same types of questions can arise across hundreds of product reviews, expert assessments take days of manual research, and inconsistency between assessments creates real organizational risk.

It’s not uncommon for experts to spend more time answering routine questions than on genuinely novel and ambiguous work where their judgment matters most. We need systems that can capture how an organization’s experts reason and make that knowledge available to everyone who needs it, so that expertise is easier to share, build on, and preserve.

We set about solving this challenge by codifying institutional intelligence into an AI agent for a specific compliance domain. The agent combines a knowledge system that acts as the organization’s “second brain,” a reasoning layer that mirrors how domain experts actually think, and an automated improvement pipeline that compounds expert effort permanently. The patterns generalize to any enterprise domain with deep specialist knowledge, whether that is finance, security, or engineering.

The Architecture at a Glance

Off-the-shelf LLMs provide a strong foundation, but they often need deeper institutional context to be fully effective in specialist domains. Without that grounding, a general purpose model has limited value given it will not be able to distinguish between what an organization could do (a summary of general information) and what it should consider doing (based on historic positions, company direction, business context, etc.). In high-stakes domains, closing this gap requires supplying the model with the organization’s own knowledge and priorities so its analysis reflects how the organization actually reasons.

The system we’ve designed has four layers, each solving a distinct problem:

These layers depend on each other. The knowledge system’s file structure makes automated editing possible. The reasoning layer’s explicit procedures make failure attribution tractable. The evaluation framework gates every change. And the improvement loop feeds back into both knowledge and reasoning. Remove any one layer and the others degrade.

Building the Organizational Second Brain

Large organizations can accumulate thousands of documents as a byproduct of expert work. It is tempting to treat those documents as organizational knowledge, but the real knowledge is implicit: how experts reason, what they prioritize, and how they resolve ambiguity. An agent that retrieves document chunks at inference time has to re-derive that reasoning from raw sources on every run, which is slow, error-prone, and inconsistent.

We make that implicit knowledge explicit ahead of time. A long-running offline process reasons through source documents and distills them into structured knowledge files – curated statements of how the organization interprets its domain, with constraints, boundaries, and routing implications made machine-readable.

Most significantly, that knowledge then forms the basis of a feedback loop that allows the agent to learn from and implement feedback from human experts without the underlying model having to be retrained.

The industry has converged on a similar idea. Andrej Karpathy’s LLM Wiki structures agent knowledge as a navigable graph of files, and Google’s Open Knowledge Format standardizes this for cross-agent interoperability. The shared insight is that knowledge should be pre-extracted, explicitly structured, and progressively disclosed rather than re-derived on every query. We extended these principles into a system where citation fidelity and institutional consistency are non-negotiable, organizing 200+ files into a strict taxonomy:

- Position files capture authoritative organizational stances: how the organization has decided to interpret a given domain question, along with its constraints, boundary conditions, and machine-actionable routing implications that tell the reasoning layer when to apply it.

- Taxonomy and vocabulary files act as an authoritative glossary for the terms the organization uses to describe its domain, such as entity types, activity categories, and classification tiers. Each is maintained as a single source of truth so the agent and the organization use language consistently.

- Routing indexes map input characteristics to the relevant positions and procedures, determining which files apply without relying on embedding similarity alone. This makes retrieval deterministic and auditable.

- Gateway files define threshold tests the agent must pass before entering an analytical domain, preventing it from applying specialized knowledge where it does not belong.

Every file declares its dependencies (depends_on) and consumers (referenced_by) in YAML frontmatter, forming a bidirectional dependency graph. When one file changes, you can trace exactly what else might be affected, which matters when the self-improvement loop proposes automated edits.

An illustration of the knowledge system: files are organized as a navigable filesystem (left), and each file’s YAML frontmatter (right) declares when it applies (the triggering scenarios) plus its dependencies and consumers, forming a bidirectional dependency graph the agent can traverse and maintain easily.

Organizing Knowledge by Density and Usage Frequency

A key architectural decision is how to partition knowledge between the curated wiki and supplementary retrieval (RAG). We split on information density and expected usage frequency.

High-density, frequently referenced sources go into the wiki: Distilled files capturing how the organization reasons, such as positions, decision frameworks, boundary examples, and strategic interpretations. The agent consults these on nearly every turn. Because they encode the organization’s evolving thinking, they need to stay current, and the wiki structure makes them easy to update, version, and validate.

Sparse, situationally relevant sources are served through semantic or lexical search (RAG): documents that matter deeply when they apply but are not needed in detail on most runs, such as detailed reference material, individual product specifications, historical decision records, and niche external knowledge. Loading all of them into the wiki would bloat the system and dilute attention.

The result is that the agent’s core reasoning is always grounded in the most refined, current organizational knowledge, while it can still reach for supporting evidence when a scenario demands it. The combination produces an organizational second brain that encodes how the organization interprets and applies information, rather than only where to find it.

Expert Reasoning via Composable Recipes

Knowledge alone is not enough. Domain experts do not simply recall facts, they follow structured methodologies: a financial analyst works through a valuation model step by step, a security engineer follows a threat modeling procedure. The challenge is capturing those methodologies in a form an LLM can execute reliably.

We solve this with composable procedures we call recipes. Where knowledge files are declarative, recipes are imperative. Each one prescribes a multi-step analytical workflow, specifying what to examine first, which knowledge to load at each step, what decision procedures to follow, and what constitutes a complete analysis.

The critical design choice is separating what the agent knows from how it reasons. Recipes reference knowledge files but contain no domain facts; knowledge files state positions but prescribe no procedures. This means:

- Adding an organizational position means adding a knowledge file and updating a routing index. No recipe changes.

- Fixing a flaw in the agent’s methodology means editing a recipe. No knowledge files change.

- Failures attribute cleanly to one layer. Was the knowledge wrong, or the procedure?

Recipes compose into pipelines, much like a head chef’s master recipe for a dinner service delegates to sub-recipes for each component (the sauce, the protein, the garnish) without containing those details itself. Our top-level routing recipe examines the input and selects which downstream recipes to invoke, each handling one analytical phase.

This is also what enables progressive disclosure. Rather than front-loading a monolithic instruction set covering every possible scenario, each recipe step carries only the instructions and knowledge relevant to that phase. Early versions used a single flat instruction file and loaded all sources via semantic search, pulling a large volume of mixed-relevance files into the context window on every run. After restructuring into recipe-driven stages, each query touches only a small, targeted subset, cutting tokens consumed per turn by around 80%. Context windows are finite and attention degrades with volume, so delivering the right instructions at the right time directly improves reasoning quality.

Keeping Humans in Control

Human experts stay in control of this system throughout. The agent accelerates and structures their work; it does not replace their judgment or their authority over the outcome.

We enforce this through two mechanisms:

Checkpoints are defined points in the analysis where the agent surfaces its intermediate reasoning for expert review before proceeding, and the expert can confirm, correct, or redirect.

Escalations trigger when the agent hits genuine ambiguity, whether from underspecified inputs or evidence that supports more than one defensible reading. Rather than forcing a resolution, it hands the question to the expert, whose choice determines the path the analysis takes.

Checkpoints and escalations serve three purposes simultaneously:

- Quality and direction control: Experts catch errors before they compound downstream, and keep the analysis on the path they consider most relevant.

- Training signal: Every correction and every escalation becomes input for the self-improvement loop.

- Trust calibration: Experts build confidence incrementally by observing the agent’s reasoning rather than just its final output, and by seeing it flag uncertainty instead of masking it.

The more consequential the decision, the more this matters, which is why we recommend keeping a human in the loop by default across domains like compliance, financial risk assessment, security review, and engineering safety.

The Self-Improvement Flywheel

We consider the self-improvement flywheel to be the most distinctive part of this system. While the structured knowledge system and composable recipes produce a system that is legible by humans and agents, testable, and modular, the number of interdependent files make manual maintenance impossible to scale. When domain experts provide feedback to an agent that feedback has to be translated into precise file edits. This process can take weeks because it r

正文由 FLUX 从来源站点 RSS 同步，内容未经改写；遇到排版缺失或需要图片、视频时请以原文为准。