postgrok

slop-code-bench

Properties3
tagsai, benchmark, evaluation, coding
typeconcept
statusseedling

2 min read

Slop Code Bench

Definition: benchmark for iterative coding ability. It gives a model a greenfield build, then asks it to add a feature to that same codebase without knowing the full problem up front.

Origin

Cited in 2026-10-04-youtube-jivko-ai-weekend (2:52 to 3:23) as the empirical evidence against AI psychosis: “This is proven empirically with benchmarks such as Slop Code Bench.” The video’s description of the mechanism, captioned:

“It gives the model a thing to build greenfield, and then it says, Cool. Now take that code base and add this feature. As models continue to work… on the same code base without knowing the whole problem up front… the code gets worse and worse over time, and… the number of failing tests just… trends up for all the models.”

Key finding, as reported in the video

  • The reported score: “I think GPT 5.5 Codex, when I ran it in May, got like a 15% on this chart” (3:19 to 3:23). The run is May 2026. An earlier version of this page said May 2024; that was an invention, corrected in log.
  • Quality degrades over time as models keep working the same codebase
  • Failing-test counts trend up across all models
  • Jivko’s gloss: “The best frontier models faceplant hard when it comes to real production software engineering. A 15% score is not something I’d consider above my skills any day of the week” (3:31 to 3:40)

unsourced

The benchmark itself was not read for this vault. Everything here is the video’s account of it. Before any public page leans on the 15% figure, the primary artifact behind the chart needs its own source page.

Significance

It marks the gap between one-prompt-awe, where a greenfield one-shot success looks like magic, and iterative work inside a large codebase, which is where production engineering actually lives. A previous filing called the awe term “one-shot automation awe.” The caption term is one prompt awe.

Contrast

Greenfield benchmarks measure the capability one prompt awe samples. Slop Code Bench, on the video’s account, measures the iterative, context-heavy work it does not.

  • ai-psychosis: the false confidence from greenfield-only evaluation
  • skill-atrophy-ai: why humans must maintain the skills models lack
  • human-ai-synergy: the different degradation story, human-AI combinations losing to the best single performer

Graph View

Backlinks