ScarfBench: IBM’s New Benchmark for AI-Driven Java Migration

IBM Research has released ScarfBench, a benchmark built to evaluate how well AI agents handle enterprise Java framework migrations. The work was published on the Hugging Face blog and targets a gap in existing coding benchmarks: most focus on greenfield code generation, not the messy, high-stakes job of moving large Java applications from one framework to another. For teams managing legacy Java systems, it signals that AI migration tooling is being taken seriously at the research level.

What happened

IBM Research published ScarfBench on the Hugging Face blog. The benchmark is designed to test AI agents specifically on enterprise Java framework migration, a task that sits well outside what most popular coding benchmarks measure.

Most existing benchmarks ask models to write new code from a prompt. ScarfBench asks a different question: can an AI agent take an existing Java application built on one framework and reliably move it to another? That is a much harder problem. It involves understanding dependencies, preserving behavior, and handling the kind of edge cases that accumulate in production code over years.

IBM Research built ScarfBench to fill that gap and to give the research community a shared measuring stick for this class of agentic coding task.

Why it matters

Enterprise Java is everywhere. Banks, insurers, logistics companies, and government agencies run enormous Java codebases, many of them tied to frameworks that are aging out or being replaced. Migrating those systems is expensive, slow, and risky when done by hand.

If AI agents can reliably handle framework migration, the cost and time savings would be substantial. But “reliably” is the operative word. Without a benchmark that reflects real migration complexity, it is hard to know whether a tool is genuinely useful or just good at simple cases.

ScarfBench gives vendors and researchers a way to make that comparison honestly. It also gives enterprise teams a reference point when evaluating AI-assisted migration tools from any vendor, including IBM itself.

The choice to publish through Hugging Face rather than a traditional academic venue is worth noting. It puts the benchmark directly in front of the ML practitioners and open-source tool builders who are most likely to use it, which tends to accelerate adoption.

Our take

Benchmarks are only as useful as the tasks they include. The interesting question with ScarfBench is whether its migration scenarios reflect the actual messiness of enterprise Java: custom configurations, internal libraries, years of accumulated workarounds, and frameworks that were never meant to be swapped out.

IBM has deep experience with enterprise Java, so there is reason to think the task design is grounded in real problems rather than tidy toy examples. But until independent teams run their own tools against it and publish results, ScarfBench is a promise more than a proof.

For agencies and development shops doing migration work, the more immediate takeaway is directional: the research community is now treating agentic code migration as a serious, measurable problem. That means commercial tools in this space will improve faster and will start making specific, testable claims rather than vague ones.

Watch for other labs and vendors to either adopt ScarfBench as a standard or publish competing benchmarks. That competition will tell you more about the state of the tooling than any single vendor demo.

What to do about it

If your team manages a Java application that needs a framework migration in the next one to two years, do two things now:

• Track which AI coding tools publish scores against ScarfBench or comparable migration benchmarks. A tool that reports specific benchmark numbers is more credible than one that only shows curated demos.
• Document your current framework dependencies and any known edge cases. That groundwork makes it easier to evaluate any AI migration tool against your actual codebase, not just the vendor’s test cases.

The tooling is not mature enough to hand a migration over to an AI agent today, but knowing which benchmarks the leading tools are measured against will help you ask better questions when it is.

Source: Hugging Face Blog