Agent test · 7 Nov 2026
Does JavaScriptAI change what coding agents build?
We gave Claude Code the same AI coding tasks in fresh projects, with and without one rule: before adding an AI library, read javascriptai.com/llms.txt and use the current, maintained package it lists for the job. On framework-sized jobs, it changed what the agent built every time. On simple ones, it made no difference.
What we found
0 → 2 of 2
Knows what's newer than the model
The MCP SDK shipped v2 in July 2026, after the model's training data. Without our list, Claude used v1 every time. With it, v2 every time.
41% less code
Uses a framework instead of writing one
Without the list, Claude hand-wrote agent memory, loops and retry code. With it, it used Mastra and LangGraph.js: 41% fewer lines on the memory agent and the crash-safe workflow.
7 → 10 of 10
The list is what does it
Same rule, our list before and after a day's update: current packages went from 7 to 10 of 10, and Mastra went from never picked to picked for the memory agent. A reminder alone isn't the effect.
Five harder tasks, side by side
Two runs per column. “Old list” is the same rule pointed at our list as it was that morning; “JavaScriptAI” is the list after that day's update.
| Task | No rule | Rule, old list | Rule, JavaScriptAI |
|---|---|---|---|
| MCP server and client | outdatedMCP SDK v1 | outdatedMCP SDK v1 | currentMCP SDK v2 (@modelcontextprotocol/server) |
| Support agent that remembers users | hand-writtenOwn memory on the Anthropic SDK | Vercel AI SDK | Mastra, with its memory and local SQLite storage |
| Crash-safe multi-step workflow | hand-writtenOwn retry and resume code | AI SDK plus own checkpoint code | LangGraph.js with SQLite checkpoints |
| Multi-agent researcher | hand-writtenOwn agent loop | Vercel AI SDK | Vercel AI SDK |
| RAG pipeline | LanceDB (one run on an old Arrow version) | Chroma, or no vector store | LanceDB |
Every run in every column passed the TypeScript typecheck. “Current” means no package deprecated on npm, replaced by its maintainer, or a major version behind, judged from npm and maintainers' announcements, never from our own list.
Code the agent wrote, per run
No ruleJavaScriptAI
Average non-blank, non-comment lines of TypeScript per run. All five tasks: 1,630 lines without the list, 1,216 with (25% less). MCP went up because the agent wired the v2 client and server packages separately.
Where it made no difference
On five simple, well-known jobs (a streaming chat function, a Gemini summary, an image classifier in the browser, basic RAG, a two-tool agent), every version picked current packages 15 times out of 15: no rule, a generic “use current packages” rule, and ours. A recent model already knows those. The value is in what changed after the model was trained, and in which framework fits a bigger job.
How we ran it
- Claude Code (Claude Opus 5.5), headless, in a fresh copy of the same small TypeScript project each run. The versions differed only in the project's
CLAUDE.md. - Every prompt ended: “Put the code in src/. Install what you need with npm and make sure npm run typecheck passes. Read API keys from environment variables, and don't run anything that calls a paid API.”
- 86 runs in all, about $35 of model use: 45 on the simple tasks across three versions, 20 on the harder tasks with the old list, 20 with the updated list, and one pilot.
- Scoring read the installed packages and the code, checked them against npm and maintainers' announcements, and re-ran the typecheck.
The five harder prompts, word for word
- MCP server and client
- Expose two functions, searchDocs(query) over docs/ and getShippingRate(country, weightKg) (stub it), as an MCP server, and write an agent that connects to that server as a client and uses its tools to answer a question. Pick the libraries yourself.
- Support agent that remembers users
- Build a customer-support agent that remembers each user's earlier conversations across restarts (persist locally, no hosted database), can look up an order with a lookupOrder(orderId) tool (stub it), and hands the conversation to a human when the user asks for one. Pick the libraries yourself.
- Crash-safe multi-step workflow
- Build a content pipeline: given an article's text, (1) extract the key facts, (2) draft a short blog post from them, (3) have a reviewer step critique the draft and revise it once. Each step must be retryable on its own, and a run must be able to resume where it stopped if the process crashes. Pick the libraries yourself.
- Multi-agent researcher
- Build a research assistant: a planner agent splits a question into sub-questions, researcher agents answer them in parallel using a searchWeb(query) tool (stub it), and a writer agent combines the answers into one report with citations. Pick the libraries yourself.
- RAG pipeline
- Build a production-style RAG pipeline over docs/: chunk the markdown by heading, embed the chunks, store them in a vector store that persists to disk, retrieve and rerank, and answer with citations to the file and heading. Add a small eval script that checks the answers to three questions. Pick the libraries yourself.
Limits
- Small: one agent and model, two runs per cell on the harder tasks. A larger run across more tasks and agents is next, and we'll publish it here whatever it shows.
- We checked packages and typechecks, not whether the code works against a live model.
- The agent was told to read the list. In real use, someone has to install the skill or paste the prompt first.
Try it on your own project
One command installs the skill for your agent. It reads our list and sends nothing about your code. How tools get listed: our standards.
npx skills add goodmeasurelabs/skills --skill javascriptai -yDownload SKILL.md