Duolingo's Two-Part Blueprint for Making AI Agents the Default

The Shared Definition Registry

At Duolingo, the recurring cost of rebuilding agent infrastructure led to a centralized approach. Each agent is defined once in a registry through an `AgentDefinition` object that specifies its name, owner, a chosen model like 'gpt-5.5', a system prompt, required MCP servers such as 'github' or 'sentry', and a structured output type. This definition becomes the reusable blueprint. Before this, teams spent weeks configuring SDKs and tooling; now, a developer uses an internal site to select options and create an agent in about 10 minutes. The registry ensures consistency, allowing any agent to be invoked from multiple surfaces—Slack, a CLI, or another workflow—without redoing foundational setup. This abstraction eliminates duplicated work and lets teams focus on agent behavior rather than infrastructure.

Temporal as the Durable Execution Layer

Duolingo's agent platform relies on Temporal as its durable execution layer. An AI agent run is modeled as a Temporal workflow, which owns the durable state and orchestration for the entire lifecycle. This is critical because agents can run for several minutes, call multiple external tools via MCP, wait for asynchronous human input, and fail in ways that require automatic retries. The workflow handles the complex setup sequence—it loads the agent definition, prepares the execution environment by cloning the necessary repository and configuring MCP servers with the proper credentials, and then invokes the agent using the selected LLM SDK. This abstraction means the workflow itself is not the agent but the manager of its production environment. Crucially, because Duolingo had already built infrastructure to trigger Temporal workflows from various entry points—Slack, internal sites, a CLI, or other Temporal workflows—any agent integrated into this system immediately becomes invokable from all those surfaces without additional integration work. The mechanism provides a consistent, reliable foundation that turns a simple agent definition into a production-ready service accessible anywhere.

Decoupling What an Agent Does from How It Runs

A core design principle of Duolingo's platform is the strict separation between an agent's behavioral definition and its execution runtime. The `AgentWorkflow` in Temporal is the mechanism that enforces this separation. It functions as a generic wrapper that, given an agent name, loads its definition from a central registry and manages the entire execution lifecycle. This includes preparing the persistent workspace with cloned code, installing dependencies, and configuring tool access. The workflow then invokes the actual agent logic using whichever LLM provider SDK is specified. This decoupling is what allows the platform to support multiple runtimes—such as the Claude Agents SDK, Codex CLI, and later the OpenAI Agents SDK—behind the same consumer-facing API. A developer defines an agent once by specifying its name, system prompt, required MCP servers, and expected output structure. From that point on, any change to the underlying runtime, model version, or SDK implementation occurs entirely behind the workflow abstraction. The interface for triggering the agent, and the agent's own definition, remain unchanged. This separation means the team can evolve the underlying models, tooling, and orchestration capabilities independently, ensuring that the hundreds of agents defined in the registry continue to function without modification.

Deterministic Evaluation Over LLM-as-Judge

Agent evaluation at Duolingo is built on deterministic grading to ensure reliability. Eval cases run the real agent against scenarios that include a repository fixture and a prompt. The `diff_assertions` grader inspects the actual git diff from the agent's run, requiring specific changes like 'import requests', excluding risky edits such as 'pytest.mark.skip', and limiting the number of changed files to prevent overreach. A `no_op_consistency` grader checks that the agent's reported outcome matches the repository state—for instance, failing if the agent claims no fix was needed but files were altered. While an optional LLM-as-judge exists for edge cases, deterministic graders form the foundation, providing objective and repeatable assessments. This approach ensures agents are evaluated on concrete actions, not just output quality.

Evals Themselves Run as Durable Workflows

The agent evaluation system is not a collection of ephemeral scripts; it executes as a full Temporal workflow suite, mirroring the durability and structure of production agent runs. This design means evals inherit the same reliability and observability. The suite workflow orchestrates the entire process: it loads a set of evaluation cases from a definition file, then spawns a separate child Temporal workflow for each individual case and for each repetition of that case. These child workflows run in parallel, enabling efficient execution of large evaluation suites. Each child workflow's task is to run the real agent against an authored scenario—using a fixture repository and a specific prompt—and then grade the output and resulting code changes. Because each eval run is a workflow, its state is persistently captured. Long-running cases can complete without interruption, and any failure is recorded explicitly in the workflow history rather than being lost to a terminated process. Once all child workflows complete, the parent suite aggregates their results, renders a detailed report, and can optionally persist the entire run to a dashboard for historical comparison. This architecture ensures that evaluations are a reliable, integrated part of the platform, providing consistent and reviewable insights into agent performance over time.

From Weeks to Ten Minutes

Before the agent platform existed, creating a production-ready agent was a complex, multi-week project. Each team had to independently select an SDK, learn its specific nuances and configuration patterns, implement the logic to clone and prepare a target repository, configure connections to multiple MCP servers, and securely wire up all necessary credentials. This foundational setup had to be repeated for every new use case, consuming significant engineering time before any agent logic could be written. The platform collapses this entire process to approximately ten minutes. A developer accesses an internal web interface where they define the agent by selecting which MCP servers it requires, choosing a model, and writing its system prompt. This definition is automatically entered into the central registry. The platform then handles every subsequent step: it prepares the execution environment, integrates the agent with the durable Temporal workflow layer, and makes it immediately invokable from any connected surface—be it Slack, a CLI, or another workflow. The agent automatically inherits all platform capabilities—durability, full observability, and integration with the evaluation suite—without any additional configuration from the developer. This shift transforms agent creation from a significant infrastructure project into a rapid, standardized process focused purely on defining the desired behavior.

The Foundation for Agent Orchestration

Because every agent on Duolingo's platform runs as a Temporal workflow, the platform inherently provides a foundation for sophisticated agent orchestration. Any workflow—whether it's an agent or another business process—can be triggered by another, and Temporal manages the state and coordination across the entire chain. This capability means an agent can be exposed as a callable tool for other agents, enabling the construction of larger, autonomous systems where multiple specialized agents collaborate on a complex task. A production example already in use is the internal Slack bot for release managers. This bot employs a composite workflow where specialized agents are orchestrated together: one agent investigates crash reports, another identifies the relevant code changes from version control, and a third synthesizes the findings into a concise summary for the team. Temporal ensures this multi-agent sequence is durable and its progress is trackable. Looking forward, the platform is focused on leveraging this orchestration foundation to create a continuous improvement loop. The next step involves automating the creation of new evaluation cases by ingesting and structuring feedback from engineers on an agent's real-world results, then feeding those structured evaluations back into the system to guide refinements. This turns user feedback directly into measurable quality improvements, using the same durable orchestration layer that powers the agents themselves.