Only 3 of 11 Autonomous Agents Finished the Job: What That Tells Us

A builder ran 11 cold AI agents against their own product with no hand-holding, and only 3 finished. The verdict: autonomous agents are still brittle by default, and the gap between "demo works" and "agent ships" is wider than most teams expect.

The setup

The HN post describes a straightforward stress test: launch 11 agents with identical goals against a real product, no scaffolding tweaks mid-run, no human checkpoints. The metric was simple: did the agent complete the assigned task end-to-end? Eight did not. The author does not attribute failure to model intelligence but to the structural problems that compound across long agentic runs: context bloat, redundant model turns, and agents that stall or loop rather than terminate cleanly.

This lands on the same day as two directly relevant efficiency releases. Nvidia published Sol-Pi, a standalone extension for Pi agents that packages four reusable mechanisms discovered through AutoResearch loops: reducing repeated model turns, compressing context replay, trimming oversized observations, and cutting unnecessary long-form generation. The stated goal is "spend less without making the agent do less useful work." Separately, a HN thread reports testing the RTK skill, a prompt engineering technique claiming 60-90% token reduction in agent runs.

Taken together, these three signals point at the same root cause.

What worked

The 3 agents that finished shared a pattern: their tasks had tight, verifiable exit conditions. When an agent knows unambiguously that it is done, it stops. One concrete example the author surfaces: agents given a task framed as "return a JSON object matching this schema" completed at a higher rate than agents given open-ended goals like "improve the onboarding flow." Schema-bounded outputs act as a natural termination signal.

Sol-Pi's context replay compression addresses a related win. In long runs, agents frequently re-read earlier context to reorient themselves. Sol-Pi batches and summarizes that replay rather than passing raw history, which reduces both token spend and the drift that comes from the model over-weighting stale context.

The failure mode is not that agents are dumb. It is that they do not know when to stop.

Where it struggled

Eight of 11 agents hit one of three failure modes:

• Loop and stall: the agent cycles through the same subtask without progress, never triggering a termination condition.
• Context overflow: mid-task context grows large enough that the agent loses coherence on its original goal.
• Premature exit: the agent decides it is done after completing only a sub-step, returning a partial result as if it were final.

The RTK technique targets the middle failure, but the HN thread notes the 60-90% token reduction claim has not been independently replicated at scale. The technique involves injecting a structured "reduce, track, kill" instruction block that tells the agent to compress working memory, track its own progress state, and hard-exit on a defined condition. It works well on constrained tasks; on open-ended ones, the "kill" condition is hard to specify in advance, which is exactly where most real product tasks live.

Sol-Pi is also Pi-specific. Teams not already on the Pi agent harness get none of the benefit without porting the mechanisms manually, which is non-trivial.

The verdict

If you are shipping agentic features today, this experiment is a useful calibration. A 27% cold-start completion rate is not a reason to abandon agents; it is a reason to instrument them properly before you put them in front of users.

For teams already on Pi, Sol-Pi is a low-friction install and worth the 30 minutes. For everyone else, the most portable lesson from all three signals is the same: define your exit condition before you define your prompt. Agents that know when they are done complete tasks. Agents that do not, loop.

The RTK skill is worth testing on your specific task distribution, but treat the 60-90% claim as a ceiling, not a floor, until you have your own numbers. Pair it with prompt engineering discipline around observation trimming and you will get closer to the real gain.

Wait on fully autonomous cold-start deployments for open-ended product tasks until you have at least one human-in-the-loop checkpoint or a verifiable intermediate exit condition baked into the scaffold.

Instrument for completion rate first; everything else is noise until your agents reliably finish.