postgrok

orlanski-2026-slopcodebench

Properties3
tagsai, benchmark, coding-agents, code-quality, arxiv
typesource
statusseedling

3 min read

Orlanski et al. 2026: SlopCodeBench

Source: Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Nicholas Roberts, Frederic Sala, Aws Albarghouthi, “SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks”, arXiv:2603.24755, submitted 2026-03-25, v2 2026-05-07 https://arxiv.org/abs/2603.24755. Snorkel AI’s listing records it as accepted to NeurIPS 2026 https://snorkel.ai/research-paper/slopcodebench-benchmarking-how-coding-agents-degrade-over-long-horizon-iterative-tasks. Code and the leaderboard live at https://www.scbench.ai; artifact DOIs 10.5281/zenodo.18405900 and 10.5281/zenodo.19257129. License on the arXiv record is CC BY 4.0.

Ingested: 2026-11-09, from the arXiv abstract page and the paper’s HTML rendering, read for the numbers the vault’s concept page had been carrying from a video relay. The filing pays the debt slop-code-bench set on 2026-10-04: the primary artifact behind the chart got its own source page.

Design

A benchmark for iterative agentic coding, the situation single-shot benchmarks hide. It has 36 problems and 196 checkpoints, and each checkpoint is a specification extension the agent must build on top of its own prior code. Internal structure is left to the agent, deliberately so: architectural decisions made at checkpoint 1 are paid for at checkpoint 6 https://arxiv.org/abs/2603.24755.

Headline numbers, from the abstract

  • 15 coding agents evaluated, open and closed models. No agent fully solves any problem end-to-end. The best agent passes 14.8% of checkpoints.
  • Two quality metrics: structural erosion (complexity concentrated in a few functions, measured as the share of overall cyclomatic complexity held by high-complexity functions) and verbosity (redundant code, flagged by AST-Grep rules and clone detection). Erosion rises across checkpoints in 77% of trajectories; verbosity in 75.5%.
  • Against 473 open-source Python repositories: agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories.
  • Explicit quality guidance (tell the agent to write clean code) cuts initial verbosity and erosion by up to a third and does not change the degradation rates. The prompt moves the start, not the slope.

Relation to the video’s account

2026-10-04-youtube-jivko-ai-weekend reported “GPT 5.5 Codex… got like a 15% on this chart” from a May 2026 personal run. The paper’s number is 14.8% of checkpoints for the best agent, so the video’s figure is the paper’s headline rounded, with the personal-run attribution approximate. The mechanism the video described, add a feature to your own codebase without knowing the whole problem, failing tests trending up, is the paper’s design. Jivko’s broader claim, frontier models faceplant at production software engineering, is a fair reading of “no agent fully solves any problem.”

The leaderboard is a moving number. A community rerun of the full benchmark in October 2026 reported the best score at 16.3% (GPT-6 Astra), against 14.8% in May, measured across the complete 36 problems https://x.com/dexhorthy/status/2115115507183685850. Independent, not peer-reviewed, and it says the ceiling is low and moving slowly.

Sources

Graph View

Backlinks