ScarfBench Tests AI Agents on Real Enterprise Java Migrations

IBM Research has released ScarfBench, a benchmark that puts AI agents through enterprise Java framework migration tasks, one of the most common and costly refactoring jobs in large engineering organizations. This is a meaningful step beyond synthetic coding puzzles toward evals that reflect real production work.

Why it matters

Most coding benchmarks measure whether a model can write a correct function from scratch. Enterprise refactoring is a different problem entirely. Framework migrations involve understanding large, tangled codebases, respecting existing architecture decisions, handling deprecated APIs, and producing changes that compile, pass tests, and do not introduce regressions across thousands of files.

ScarfBench operationalizes that complexity into a structured eval. For teams deciding whether to trust an agent with a Spring Boot upgrade or a Jakarta EE migration, a score on a general benchmark tells you almost nothing. A score on ScarfBench is at least asking the right question.

"The gap between 'can write a loop' and 'can migrate a production codebase' is where most agent deployments quietly fail."

This also lands at a moment when agentic tooling is proliferating fast. OpenClaw just shipped on Android and iOS, model-agnostic agent frameworks like Agent Sessions are emerging as Claude alternatives, and Hugging Face and Cerebras are wiring capable models like Gemma 4 into real-time agentic pipelines. The infrastructure is outpacing the evaluation layer. ScarfBench starts to close that gap for one high-value vertical.

What changes in practice

• Teams evaluating coding agents for refactoring work now have a domain-specific benchmark to reference, rather than extrapolating from HumanEval or SWE-bench scores.
• Vendors pitching AI-assisted migration tooling will face sharper scrutiny: ScarfBench gives procurement teams a concrete question to ask.
• The benchmark establishes a reproducible task format that other enterprise verticals (Python framework migrations, cloud SDK upgrades) can clone and adapt.
• Agents that score well here are more likely to handle the ambiguity and scale of real migration projects, not just isolated file edits.

How to use it

1. Run your current coding agent against ScarfBench before committing it to any migration project. Treat a low score as a hard stop, not a yellow flag.
2. Use the benchmark tasks as fine-tuning signal if you are customizing a base model for internal refactoring workflows. The task structure maps closely to what your engineers actually do.
3. Reference ScarfBench scores in vendor evaluations. Ask any AI-assisted migration tool vendor for their numbers. If they do not have them, that is informative.
4. Pair the benchmark with a human review gate on the first few migrations. ScarfBench measures task completion, not code review quality or team-specific conventions.

Domain-specific benchmarks like ScarfBench are how the industry will eventually separate agents that are useful in production from agents that are impressive in demos.