A coding agent can have access to the right documentation and still ignore it. Vercel measured that failure directly: its default skill setup passed 53% of a Next.js evaluation, exactly the same as having no documentation. A short AGENTS.md file with a compressed documentation index reached 100%.

That sounds like a victory for putting everything in one markdown file. It is not. A separate 2026 study found that repository context files increased inference cost by over 20% on average and did not generally improve task success. The useful answer sits between those results: keep persistent instructions small, make retrieval obvious, and test the file against real tasks before you decide it belongs in every session.

Vercel's reported pass rates for four coding-agent setups

The setup that worked

Vercel was trying to solve a specific problem. Next.js 16 introduced APIs such as use cache, connection(), forbidden(), and unauthorized() that may not exist in a model's training data. An agent can write perfectly plausible code using an older pattern, then fail when the project builds. Version-matched documentation matters more than generic advice about writing clean React.

The team tested two ways to expose that documentation. A skill packaged the docs and was supposed to be invoked when the agent recognized a Next.js task. AGENTS.md stayed in the repository root and was visible on every turn. The default skill was never invoked in 56% of evaluation cases. Its pass rate was 53%, with no improvement over the no-documentation baseline. Adding an explicit instruction to explore the project and then invoke the skill pushed the result to 79%.

The best result removed the decision point. Vercel generated a compressed index in AGENTS.md that pointed at version-matched files in .next-docs/. The index was about 8KB after compression, down 80% from the initial 40KB injection, and the reported pass rate stayed at 100%. The file did not contain every page of documentation. It told the agent where the relevant pages lived and told it to prefer retrieved information over its pretraining for Next.js tasks.

That is a much better pattern than dumping a giant rulebook into the root of a repository. A good file has three jobs:

  • State rules that apply to every task, such as how to run checks or which directory boundaries matter.
  • Point to the small set of reference files an agent should read for a particular framework or subsystem.
  • Tell the agent when retrieval wins over memory, especially for fast-moving APIs.

For a new project, start with a file that fits on one screen. Add commands that actually work, the test command, the build command, the package manager, and any rule that would make a seemingly valid change wrong. Link to deeper material instead of pasting it.

Then run a small evaluation. Pick five tasks that resemble your real work: one feature, one bug, one refactor, one test change, and one task involving a version-sensitive API. Run them with the file and without it. Record build, lint, test, and review failures. You do not need a research lab. You do need a baseline, the same prompts, and enough repetitions to notice when an improvement was luck.

This fits well with the advice in the long-prompt problem for coding agents. The fix is not to keep adding prose. The fix is to move durable context into a place the harness can load predictably, while keeping expensive reference material retrievable. If several agents are working at once, pair the file with a local supervisor for parallel agents so the repository rules do not become an excuse to let every worker write wherever it wants.

Where it breaks

The Vercel result is useful, but it is not a universal law. The evaluation focused on Next.js 16 APIs and a single agent setup. A file that contains exactly the missing framework knowledge can look spectacular in that test. It tells us less about a repository with many unrelated subsystems, several agent products, or a large set of changing instructions.

The Hacker News discussion made that objection plainly. Several developers pointed out that the AGENTS.md index is similar to a well-designed skill, with fewer decisions between the model and the relevant docs. Others argued that always-visible context wastes tokens once a project accumulates several workflows. Both criticisms are fair. Passive context wins when the model repeatedly misses the same lookup. Skills win when the reference is large, specialized, or rarely needed.

The broader arXiv study is the guardrail. Across established SWE-bench tasks and issues from repositories with developer-written context files, the researchers found no general task-success improvement and an average inference-cost increase above 20%. They also found that agents followed explicit instructions better than repository overviews. That suggests a practical split: put policies and non-standard conventions in AGENTS.md, but be suspicious of long tours of the codebase that merely describe what a competent search can discover.

Use this decision rule:

  • Put it in AGENTS.md if every coding task needs it, it is short, and getting it wrong causes repeated failures.
  • Put it in a skill if it describes a procedure someone invokes for a particular job, such as a migration or release workflow.
  • Put it in referenced files if it is detailed framework documentation, a long API catalog, or material used only occasionally.
  • Use a hybrid when the agent needs a persistent instruction telling it which skill or reference to load.

The failure mode to watch is context creep. A file starts at 2KB, grows to 12KB after a few incidents, then becomes a second system prompt. At that point, measure again. If the agent is spending tokens rereading rules while still missing the code change, the file is making the system worse.

A repeatable test before you keep it

Create two branches or two clean workspaces. In the first, use the repository without the new file. In the second, add the file and any reference index. Give the same agent the same task prompts. Keep tool permissions and model settings constant. Run the project checks after each attempt.

Score the outcome with facts rather than vibes: did it compile, did the relevant tests pass, did it touch unrelated files, and did the reviewer find a rule violation? Also record input tokens or request cost if your harness exposes them. A 10-point quality gain that costs 30% more inference may be worth it for a production migration and wasteful for autocomplete-sized tasks.

If the file helps, keep the smallest version that preserves the result. Delete rules that merely restate standard style. Replace long explanations with a command or a path. When a rule prevents a real failure, add a short example and the check that catches a violation. Review the file when the framework version changes, not every time someone has a vague feeling that the agent is being unhelpful.

The practical lesson is less glamorous than the benchmark: agents do better when the information they need is easy to find and hard to ignore. AGENTS.md is a good front door. It should not become the entire house.

Sources