What SlopCodeBench Measures — and How to Gate Against It
SlopCodeBench is an independent research benchmark. aislop is a scanner. The paper named the erosion. The CLI scores the shapes that keep showing up.
SlopCodeBench (arXiv 2603.24755) is a benchmark of long-horizon coding-agent work. Agents implement a problem, then extend their own solution as new requirements arrive. Evaluation is black-box: a CLI or API contract, no prescribed architecture. Early design choices compound.
It is not aislop. It is not this site’s unpublished agent leaderboard. The paper measures degradation across checkpoints. aislop scores named residue in a repository you already have.
The benchmark asks whether the agent still produces inheritably clean code. A gate asks whether this change contains the shapes that made the last change worse.
What the paper actually tracks
The public abstract describes 36 problems and 196 checkpoints. Agents pass individual checkpoints more often than they survive the full trajectory. Quality erodes as the run continues: structural complexity concentrates, and verbosity — redundant or duplicated code — rises. Compared with human-maintained Python repositories, agent output is more verbose and more eroded.
Two signals matter for a working team:
- Verbosity. Comment padding, duplicated helpers, leftover branches from earlier attempts.
- Structural erosion. Complexity mass collecting in a few oversized functions as the agent appends instead of refactoring.
Tests can stay green while the file becomes harder to change. The next prompt pays the tax.
Official site: scbench.ai. Code: SprocketLab/slop-code-bench.
Paper signal to aislop rule
You should not rerun SlopCodeBench in CI. It is a research protocol: pinned tasks, hidden tests, multi-checkpoint trajectories, published raw output. A merge check needs a cheaper, stable signal on the diff in front of you.
The transferable part is the residue. These are the shapes aislop already names:
- Comment padding → narrative comment, trivial comment
- Leftover helpers from earlier attempts → thin wrapper, empty function, unreachable code
- Silent “it still works” paths → swallowed exception
- Type-system escape hatches → unsafe type assertion
Python verbosity checks drawn from the paper shipped in aislop v0.9.4. Run npx aislop scan. The same file produces the same score.
A Monday-morning gate
If the paper convinced you the erosion is real, do not start by reproducing 196 checkpoints. Start with the change that is about to merge:
- Run npx aislop scan on the branch. Read the named findings, not the score in isolation.
- Fix the mechanical hits (aislop fix) and send the rest back to the agent that wrote them.
- Add the GitHub Action from quality gates for AI coding. Pin the CLI version. Start the threshold at the current score so you do not block on legacy debt.
- Keep human review for intent, architecture, and risk. The gate is the residue check, not a substitute for that.
Where to go next
- Named AI slop code examples — flagged shape and clean shape per rule.
- aislop’s own agent benchmark — same tasks, deterministic score, not yet published.
- What aislop is — the CLI, not the Reddit and not the coin.
Frequently asked questions
What is SlopCodeBench?
SlopCodeBench is an academic benchmark, published as arXiv 2603.24755, that measures how coding agents degrade when they repeatedly extend their own solutions under changing requirements. It is hosted at scbench.ai and is not a product from scanaislop.
Is SlopCodeBench made by aislop?
No. SlopCodeBench is independent research. aislop reads the paper, turns repeatable residue into deterministic rules, and scores those shapes on a real repository. The two projects answer different questions.
How do I gate against what SlopCodeBench measures?
You cannot rerun the full benchmark on every pull request. You can detect the residue it describes: verbosity, duplicated helpers, comment padding, and structural erosion. Run npx aislop scan locally and in CI so those shapes fail the same way every time.
What is slopcodebench arxiv 2603.24755?
It is the identifier for the paper SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks. Search that ID and you should land on arXiv, scbench.ai, or this explainer — not on a hosted quality-gate product.