Agent evals for Next.js. Each eval is a small Next.js app + a prompt + assertions. We run the prompt through a coding agent in a sandbox and check what it wrote.

The point: find places where agents get Next.js wrong because their training data is stale, then fix it by shipping better docs in the next package itself.

The runner is @vercel/agent-eval. It spins up a sandbox (Vercel or local Docker), copies the fixture in, runs the coding agent against PROMPT.md, then executes EVAL.ts as a vitest file against whatever the agent wrote. The PROMPT.md / EVAL.ts / fixture-dir convention you'll see below is that package's convention — see its README for the full spec.

run-evals.js is a thin wrapper around it: pack the local next build into a tarball, generate the configured experiments, then invoke agent-eval. The two default experiments (baseline and agents-md) differ only in whether they drop an AGENTS.md pointing at the bundled docs. Everything from "spawn sandbox" onward is @vercel/agent-eval's job.

Vercel employees: request access to the vercel-labs team in Lumos, then:

# Vercel CLI, if you don't have it: npm i -g vercel
vc link       # at repo root, pick vercel-labs team
vc env pull   # writes .env.local to repo root

External contributors can run the same evals in local Docker with their own API key — see Running without Vercel sandbox access.

Copy an existing fixture. Take the next free number — gaps are fine.

cp -r evals/evals/agent-034-async-cookies evals/evals/agent-042-your-thing

Then edit three files:

PROMPT.md — what you'd type into the agent. Write it like a real user would: describe the symptom or goal, not the API. "Navigating from /a to /b is slow, fix it" is a good prompt. "Use instant" is not — you're testing whether the agent understands the feature well enough to reach for it, not whether it can pattern-match a name you handed it.

EVAL.ts — vitest assertions against files the agent wrote. Regex the source, don't run it.

import { expect, test } from 'vitest'
import { readFileSync } from 'fs'
import { join } from 'path'

const page = readFileSync(join(process.cwd(), 'app/page.tsx'), 'utf-8')

test('exports instant', () => {
  expect(page).toMatch(/export const instant\b/)
})

app/ (or pages/) — the starting state. Give the agent something to edit, not a blank slate.

package.json needs a build script. next.config.ts and tsconfig.json stay unless your feature requires specific config.

If the agent needs prepared runtime state, add an eval:setup script. It runs after Next.js is installed and before the agent starts.

pnpm eval agent-042-your-thing

This runs the two default variants in parallel and prints pass/fail for each:

✗ baseline/agent-042-your-thing   (81s)
✓ agents-md/agent-042-your-thing  (200s)

agents-md drops an AGENTS.md into the sandbox telling the agent to check node_modules/next/dist/docs/ first. baseline doesn't. That's the whole difference — same prompt, same model, one extra file. If agents-md passes and baseline doesn't, the bundled docs are doing their job.

Docs can link to a canonical skill, but an unmerged skill revision isn't part of the next package tarball. To compare the current checkout's skill with the baseline and bundled-docs variants, add the fixture to evals/eval.config.json:

{
  "agent-046-adopt-partial-prefetching": {
    "skills": ["next-partial-prefetching-adoption"],
    "timeout": 1800
  }
}

The runner then adds a third skills variant for that fixture. It installs the listed directories from the local skills/ folder before the coding agent starts, while keeping the prompt, app, and assertions identical. It does not also inject the agents-md instruction: the skill treatment measures whether the skill itself leads the agent to the canonical bundled guide. The optional timeout lets end-to-end workflows run longer than the 12-minute default. Fixtures without an entry continue to run only baseline and agents-md.

A run takes ~2–5 min. To validate a fixture without executing:

pnpm eval agent-042-your-thing --dry

Full transcripts land in evals/results/<variant>/<timestamp>/<eval>/run-1/. Grep transcript-raw.jsonl to see exactly what the agent did.

pnpm eval packs packages/next/dist/ into a tarball and ships that to the sandbox. It does not build. If you changed packages/next/src/** or docs/**, run pnpm --filter=next build first or the sandbox will see stale code. If you only changed fixture files, no rebuild is needed.

1. Write the fixture. PROMPT.md describes a user-facing problem. EVAL.ts asserts the API you expect the agent to reach for.

2. Build Next.js. pnpm build. The eval runner packs whatever is already in dist/ — it won't build for you.

3. Run it. pnpm eval <name>. If the feature isn't in the agent's training data and isn't documented in dist/docs/, both variants fail. That's the expected starting point for a new feature.

4. Write the doc. Add an .mdx under docs/. Use version: draft in the frontmatter to keep it off nextjs.org while still bundling it into the package.

5. Build again. New doc needs to land in dist/docs/ before the next pack sees it.

6. Run it again. baseline should still fail; agents-md should find the new doc and pass. Baseline staying red while agents-md flips green tells you the doc did it, not run-to-run noise.

7. Commit the eval and the doc together. The full suite gets pulled by the external benchmark runner and published to nextjs.org/evals. Keeping the fixture alongside the doc it validates means that score tracks over time as both the docs and the models change.

evals/
├── eval.config.json # optional skill and timeout settings by fixture
├── evals/agent-*/   # fixtures
├── lib/setup.ts     # uploads tarball, writes AGENTS.md (shared by all evals)
├── experiments/     # generated per-run, gitignored
├── .tarballs/       # packed next, gitignored
└── results/         # transcripts + outputs, gitignored

Sandbox tokens live in .env.local at the repo root (from vc env pull).

If you don't have Vercel credentials, @vercel/agent-eval falls back to local Docker — see its direct API keys docs for the full list of supported env vars. Have Docker running and provide your own model key in .env.local at the repo root:

ANTHROPIC_API_KEY=sk-ant-...

Then run pnpm eval <name> as normal. Docker pulls node:24-slim on first run. Tarball packing, both variants, and the results layout are identical to the remote path — run-evals.js doesn't know or care which sandbox backend got picked.