Loop Engineering: How Autoresearch Turns AI Agents Into Self-Improving Systems

On March 7, 2026, Andrej Karpathy released autoresearch, an open-source MIT-licensed repository of roughly 630 lines of code that runs AI-driven experiments in a continuous loop without human intervention between cycles. Over two days it completed about 700 experiments on Karpathy's own GPT-2 training code and found 20 real improvements, cutting training time by 11%. A follow-up research paper, Bilevel Autoresearch, adds a second loop on top that rewrites the first loop's search logic at runtime, producing a 5x larger improvement on the same benchmark.

What happened

Detail	Fact
Release date	March 7, 2026
Repository size	~630 lines of code, 3 core files
GitHub stars	~90,000
Experiments run (Karpathy test)	~700 over two days
Improvements kept	20
Training time reduction	11% (2.02 hours to 1.80 hours)
Shopify overnight test	19% improvement, 37 experiments
Bilevel val_bpb gain vs single loop	5x (-0.045 vs -0.009)

Karpathy pointed autoresearch at his already-optimized nanochat GPT-2 training code. The loop edited only train.py, which holds the GPT model, optimizer (Muon and AdamW), and the training loop itself. It was blocked from touching prepare.py, the evaluation utilities. That hard boundary matters: the agent cannot make the test easier instead of making the model better.

Each cycle ran one experiment. The agent read the code, proposed a change, trained for five minutes, then kept or rolled back based on val_bpb (validation bits per byte, a measure of prediction quality where lower is better). That budget yields roughly 12 experiments per hour, so about 100 run overnight. One specific find: a QK-Norm implementation was missing a scalar multiplier, leaving attention spread too thinly across heads. Humans generally tire after about a dozen experiments. The loop does not.

Shopify CEO Tobi Lütke ran the same tool overnight on an internal model and reported a 19% improvement after 37 experiments. Karpathy’s summary: if you have an objective metric, you are the bottleneck.

What separates a real loop from a chatbot on repeat?

Three components are required. Without all three, the system is not a loop, it is a very slow chat session.

• A verifier. Something automatic grades each attempt. In autoresearch that is val_bpb via prepare.py. In a software loop it could be a passing test suite or a clean build. Without a verifier, the agent simply agrees with itself.
• State. A log of what was tried, what failed, and what remains. A side file lets the next run resume rather than restart from scratch.
• A stop condition. The loop halts when the goal is met or after N attempts. Without this, compute costs spiral.

AI engineering teams now build loops from five reusable pieces: an automation trigger, a skill file (project knowledge stored in markdown, read on every run), sub-agents that split writing from reviewing, connectors into real tools like issue trackers or Slack, and the verifier as the final gate. According to the source, Claude Code and Codex now ship all five of these components.

Prompt vs loop vs bilevel loop: how they compare

Aspect	One-shot prompt	Karpathy loop (autoresearch)	Bilevel Autoresearch
You define	Each step	The goal, once	The goal, once
Who iterates	You	Inner agent	Inner + outer agent
Verifier	You, manually	prepare.py (val_bpb)	Same metric, two levels
State	Chat only	Experiment log	Log plus injected code
Reported result	Varies	700 runs, 20 fixes, 11% speedup	5x larger val_bpb drop

What is Bilevel Autoresearch?

Researchers asked a pointed follow-up: if autoresearch is a research method, can you autoresearch the autoresearch itself? The paper “Bilevel Autoresearch: Meta-Autoresearching Itself” answers yes.

The inner loop is identical to Karpathy’s original: propose, train, evaluate, keep or discard. The outer loop watches the inner loop, reads its code and execution traces, identifies where the search keeps stalling, writes new Python mechanisms, injects them at runtime, and reruns the inner loop with those changes in place.

The design has three levels. Level 1 runs the base loop. Level 1.5 tunes search parameters every five iterations. Level 2 generates new mechanisms through a four-round session. The experiments used an RTX 5090 (32 GB) with a 300-second budget per run. Crucially, both loops used the same underlying LLM. The gain came from architecture, not a smarter model. The problem the outer loop solved: the inner loop kept returning to the same priors even after they stopped producing improvements. The outer loop forced unfamiliar exploration and broke those patterns.

For teams already exploring AI integration in their workflows, this layered approach signals where autonomous tooling is heading: systems that self-modify their own search strategies, not just their outputs.

Where does this apply outside of ML research?

The loop pattern transfers beyond model pretraining whenever work can be graded automatically:

• Software refactoring: loop until tests, types, and the build all pass.
• Content production: loop until every rubric score clears a defined threshold.
• Data pipelines: loop until schema validation holds end-to-end.
• Hyperparameter search: loop until a target validation metric is reached.

Each case shares one trait: an automatic gate that can reject the work without a human in the seat. If you cannot define that gate, a loop will not help. You need a prompt, or a better-specified problem. We covered related developments in OpenAI’s agentic coding efficiency gains with GPT-5.6 Sol, where the same pressure toward autonomous iteration is visible.

Our take

The headline numbers are real, but they come with a context most coverage glosses over. Karpathy ran this on code he had already spent considerable time optimizing. The 20 improvements out of 700 attempts is a 2.9% hit rate. That is not a criticism: it is exactly what makes the loop valuable. No human team runs 700 variants of anything. The loop earns its compute cost precisely because humans would stop at experiment 12.

The harder lesson is the verifier requirement. Most business problems do not have a clean val_bpb equivalent sitting there ready to go. Defining a reliable, automated quality signal is often the actual work. Once you have it, the loop machinery is almost secondary.

For practical use, start small. Pick one workflow where you already measure output quality automatically, whether that is a passing test suite, a structured scoring rubric, or a conversion metric. Wire a loop around it. If you need help scoping that kind of workflow automation, that is where the real design decisions live, not in picking the fanciest agent framework.

The Bilevel result is the more interesting research signal. A 5x improvement from the same model, same hardware, purely by letting the system rewrite its own search logic, is the kind of result that should make any team think carefully about how they structure iteration today.

Source: Marktechpost