LLM Coding Benchmarks (Oct 2026): Which Ones Still Count

As of 22 October 2026, the coding benchmarks that still separate frontier models are SWE-bench Pro, Terminal-Bench 4.0, Cognition’s FrontierCode 1.1 and Cursor’s CursorBench 4.0, plus fresh-task sets like SWE-rebench. SWE-bench Verified is done: Vals AI archived it at a 97.00% top score, and Anthropic’s Claude Opus 5.5 launch didn’t report any SWE-bench variant.

The catch: none of these boards grades what your users see. And the same model can post four different scores on one benchmark depending on who ran it. Both problems are below, with a dated status board and the five checks Startrise runs before trusting any number.

The coding benchmarks that matter, at a glance

As of 22 October 2026, four boards are current for frontier coding models (SWE-bench Pro, Terminal-Bench 4.0, FrontierCode 1.1 and CursorBench 4.0), while SWE-bench Verified is saturated and Aider polyglot is stale. Every row links to the benchmark’s own site or an independent evaluator.

The move in 2026 has been away from “does the patch pass the tests” and toward long-horizon agent sessions and rubrics that ask whether a maintainer would merge the change. Benchmarks that only check the first question are the ones going stale.

What each benchmark actually measures

Each coding benchmark tests a different kind of work: repo issue-fixing, terminal agent sessions, merge-readiness, IDE agent tasks or competition problems. A score only tells you about work that looks like its tasks.

SWE-bench Pro

Scale’s SWE-bench Pro has 1,865 tasks across 41 repositories. The 731 public tasks come from copyleft repos, 276 private tasks from 18 proprietary startup codebases, and 858 are held out. As of 22 October 2026 the public board’s top row is Muse Spark 1.1 at 61.50 ± 3.10, then gpt-5.4 (xHigh) at 59.10 ± 3.56, both on mini-swe-agent.

There are no GPT-6 Astra or Claude Opus 5.5 rows, and the page prose still says top models score “around 23%”. The verifier has known error rates too (see the contamination section).

Terminal-Bench 4.0

Terminal-Bench 4.0 shipped on 28 August 2026, billed as “calibrating task resources, fixing tasks, and removing saturated tasks”. Vals AI describes it as “a fresh set of 66 community-contributed, maintainer-reviewed tasks”. An agent gets a shell and a goal and has to finish the job.

Version 2.1 came out in May and 3.0 in July. Never compare scores across versions. And read the agent column: the same model in Claude Code, Codex or a neutral harness is a different entry. The full comparison is in the “four numbers” section below.

FrontierCode 1.1 (Cognition)

FrontierCode is Cognition’s benchmark: 150 tasks, with a Main subset of the 100 hardest. Each patch is graded against held-out tests and a maintainer-authored rubric covering correctness, regression safety, build and style cleanliness, and project conventions. Failing a blocker criterion scores zero. It asks “would this be merged”, not just “does it pass”.

Anthropic’s vendor-reported table puts Claude Opus 5.5 at 54.4% on Main and GPT-6 Astra at 53.3%. Those are vendor numbers. We couldn’t load Cognition’s leaderboard (it renders client-side), so there’s no independent FrontierCode row here.

CursorBench 4.0

CursorBench 4.0 is Cursor’s own agent benchmark: “ambiguous, multi-file tasks from real Cursor sessions”, introduced 10 September 2026. As of 22 October 2026 the top row is Opus 5.5 at Max effort, 57.8%, with Sonnet 5.5 Max at 55.5%.

Two caveats from the page itself. It warns that “small differences in scores may not be statistically meaningful”. And it lists Cursor’s in-house Composer 2.5 at 27.7%. That’s a vendor-run board ranking the vendor’s own model, so read it accordingly. Effort moves the score more than the model does at the top: Opus 5.5 runs 57.8% at Max and 43.7% at Low, while the gap from Opus 5.5 Max to Sonnet 5.5 Max is 2.3 points.

Fresh-task sets: SWE-rebench and LiveCodeBench

Both control contamination by date. SWE-rebench’s current window is 15 May to 1 July 2026: 111 problems from 65 repos, led by Fable 5 [high] at 64.5% ± 1.41%. That window predates the September models, so Opus 5.5 and GPT-6 Astra aren’t on it yet.

LiveCodeBench release_v6 covers problems released May 2023 to April 2025, 1,055 in total. You can score by date window to skip problems from before a model’s training cutoff. It’s competition-style work, not repo work.

WebDev Arena

WebDev Arena has two models build the same web app from a user’s prompt, the user votes, and a Bradley-Terry model turns votes into scores. About 18% of votes are “both bad”. Blog du Modérateur’s 5 October 2026 snapshot had Claude Opus 5.5 Max on top at 1,815.

The same article has the fine print: the leader had just over 2,000 votes against nearly 17,000 for Claude Opus 5 Max, and a vote judges one request, not maintainability. It’s a preference board, not a recommendation.

Which benchmarks are saturated, retired or stale?

As of October 2026, SWE-bench Verified is saturated and archived, the Aider polyglot board hasn’t been updated since 20 November 2025, and HumanEval and MBPP no longer separate frontier models.

SWE-bench Verified. Vals AI has archived it: “Since performance on this benchmark has saturated, we no longer run this benchmark on new model releases.” Its top row is Claude Opus 5 at 97.00% (updated 1 September 2026). In April, Anthropic reported Claude Mythos Preview at 93.9% on Project Glasswing. SWE-bench co-creator Ofir Press wrote in a Hacker News comment that Verified was “now saturated at 93.9% (congrats Anthropic)”, pointing to SWE-bench Multilingual and Multimodal as still unsaturated. OpenAI publicly abandoned Verified in February.

Aider polyglot. The board says “last updated November 20, 2025”, and its top row is gpt-5 (high) at 88.0%. Good history, no use for a 2026 shortlist.

HumanEval and MBPP. Function-level tests from an earlier era. They won’t separate a frontier shortlist, and we don’t quote a saturation figure because we couldn’t find a primary source for one.

So check the board’s last-updated date against the model’s release date. Our model release tracker lists ship dates, each sourced to the official announcement.

Contamination, broken tests and cheating: what 2026 found

In 2026, four separate findings showed that coding benchmark scores can be inflated by training-data contamination, verifiers that misgrade patches, or agents that hack the task instead of solving it.

OpenAI’s audit of Verified. On Latent Space (23 February 2026), OpenAI’s Frontier Evals team said frontier models could reproduce gold patches or problem statements “just from SWE-Bench Verified Task ID alone”. Of 138 problematic problems they reviewed, 49 had tests “too narrowly defined” and 26 had tests “too wide”.

Fresh tasks expose inflation. The SWE-rebench paper found that “performance of some language models might be inflated due to contamination issues.”

SWE-bench Pro’s verifier misgrades. Datacurve’s DeepSWE post (May 2026) reports SWE-bench Pro’s verifier with 8.5% false positives (wrong implementations accepted) and 24.0% false negatives (correct ones rejected), against 0.3% and 1.1% on DeepSWE. It also found Opus 4.7 recovering gold solutions from the repo’s .git history. Their own caveat: verdicts come from an LLM analyzer, not human review. The Neuron has a readable summary.

Hackable tasks. A June 2026 paper, Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops, audited 1,968 tasks across five terminal-agent benchmarks and found 323 (16%) hackable. Terminal-Bench’s own integrity update says the community found “multiple instances of cheating and reward hacking”, now requires ATIF trajectories for all passing trials, and gives reward hacking a score of 0.

None of this makes benchmarks useless. It makes small gaps meaningless.

One model, four numbers: how to read a coding leaderboard

The same model can score anywhere from 59.6% to 66.4% on one benchmark version, depending on who ran it and how. Here’s Claude Opus 5.5 on Terminal-Bench 4.0, from four sources, all checked 22 October 2026.

That’s a 6.8-point spread for one model on one benchmark version. Order isn’t stable either. Vals and Artificial Analysis use the same harness and both average three runs, yet Vals has Opus 5.5 ahead of Sonnet 5.5 (65.15% vs 64.14%) while Artificial Analysis has Sonnet 5.5 ahead (63.6% vs 59.6%).

Even Anthropic says this. Its launch post notes that “benchmark margins have become a less reliable guide to real-world differences” at this level. Harness effects cut both ways, too: our Muse Spark 1.2 run found a strong model losing a single-shot brief to a bug a linter would catch, the kind of mistake an agent loop that fixes its own errors can recover from.

Before you act on any coding score, run five checks:

1. Who reported it? Vendor launch post, or an independent evaluator.
2. Is the harness named? Claude Code, Codex, mini-swe-agent and a vendor’s internal scaffold are different tests.
3. Is the effort or budget named? On CursorBench, effort alone moves Opus 5.5 by 14.1 points.
4. Are repeats or error bars shown? A ±3 band swallows most headline gaps.
5. Is the version current, and is the board dated after the model shipped? If not, the model you care about may not be on it.

Which benchmark matches your work?

Pick the board that looks like your workload, then put its number through the five checks above.

If the widget doesn’t load, here’s the short version:

• Fixing issues in an existing repo: read SWE-bench Pro first, then FrontierCode 1.1 for merge-readiness.
• Long agent sessions in a terminal: read Terminal-Bench 4.0, in the agent column you’ll actually run, then SWE-rebench (check its window date).
• Multi-file work inside an IDE agent: read CursorBench 4.0, compared within version only, then Terminal-Bench 4.0.

The gap: no public coding benchmark grades what users see

No major public coding benchmark grades whether the interface a model builds looks or works right for users. Pass/fail tests check behaviour. They don’t check whether a page looks right, reads well or meets accessibility standards. The closest public test is SWE-bench Multimodal: 617 tasks from 17 JavaScript libraries, each with at least one image in the problem statement or tests. It measures visual bug-fixing, and its authors found “top-performing SWE-bench systems struggle” with it. WebDev Arena measures preference on one request.

OpenAI’s evaluators point the same way. Their list of where coding evals must go includes “code quality and maintainability”, “real world product building” and “human-intensive evaluation” (Latent Space).

That gap is why we built the Startrise LLM Benchmark. Every model builds the same 12 frontends single-shot, each build is gated in a real browser, and a cross-lab panel judges them blind. So far that’s 30 models from 9 labs across 5 runs, 353 deliverables in all. How it works, bugs included, is written up separately. So are which model to pick for frontend work and why we audited the AI judges for bias before trusting their design scores.

Before you standardize a team on a model

Public boards build the shortlist. Your repo makes the decision.

Take 20 to 50 tickets your team actually closed. Run each shortlisted model in the harness the team will use, at the effort level you’ll pay for. Score on merge-readiness, not just tests passing: would your reviewer approve this diff as-is? Record cost per task next to the pass rate. Unlike a public board, that eval is about your code, and you can re-run it when the next model ships.

That’s the work in our AI Enablement engagements: we roll out Claude Code or Cursor, configure it for your repos, and set the standards and review gates for what AI-written code needs before it merges. From $5,000, 2–4 weeks.

If an AI build has already stalled and nobody can say whether a change helped, AI Project Rescue adds an evaluation harness so “it seems better” becomes a measured number (from $3,500). If AI-written code is passing tests but isn’t mergeable, that’s Vibe Code Cleanup. The rest of our AI automation work is on one page. Or tell us what you’re choosing between.