Cheddar-bench
unsupervised benchmark for coding agents
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
I built a small benchmark to test CLI coding agents on blind bug detection.A challenger agent injects bugs and writes ground truth (`bugs.json`). A different reviewer agent audits the repo without seeing ground truth, and an LLM matcher scores bug-to-finding assignments.Current run: 50 repos, 150 challenges, 450 reviews, 2,603 injected bugs.Weighted detection: Claude 58.05%, Codex 37.84%, Gemini 27.81%.LLM-judge benchmarks are easy to get wrong, so I’d really appreciate critical feedback on benchmark fairness, scoring/matching methodology, and obvious failure modes I’m missing.Full dataset is linked in the docs.
HN · launched Feb 22, 2026 · 9 upvotes (top 51%) · 0 views
Niche: ai agent evaluation platforms · 182 launches, 8 in the last 7 days ▼ -50% · Explore this niche →
Category and tags
- Horizontal
- Observability & eval
- Developer
- AI-native
- Hobby / open-source project
Visit website ↗ Source listing ↗
More like this: all 182 launches in ai agent evaluation platforms →
Could you build this?
Yes This is a benchmarking harness that scripts LLM API calls to inject code changes, prompts another agent to find them, and uses an LLM-as-a-judge matcher to calculate metrics.
Discussion
No comments on this launch.
Competitors
Other products that read as similar to this one : 1030 launches clear the similarity bar, closest 8 shown.
Attention rank: #509 of 1031 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 114 days after the earliest competitor.
- Parsimony · hn · 2026-10-13 · 77 upvotes · similarity 0.53
- Pencil Puzzle Bench · hn · 2026-03-03 · 5 upvotes · similarity 0.51
- OpenBenchmarks · hn · 2026-07-11 · 6 upvotes · similarity 0.50
- Cua-Bench · hn · 2026-01-26 · 40 upvotes · similarity 0.50
- LLM agents that write Python to analyze execution traces at scale · hn · 2026-03-07 · 5 upvotes · similarity 0.49
- RepoGym · ph · 2026-09-23 · 1 upvotes · similarity 0.48
- Benchmark: AI doesn't find bugs unless you tell it what's wrong · hn · 2026-10-01 · 5 upvotes · similarity 0.48
- reuF2F · github · 2026-10-18 · 22 upvotes · similarity 0.48
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.