New eval from SWE-bench team evalutes LMs based on goals not tickets
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
Current evals test LMs on tasks: "fix this bug," "write a test"But we code to achieve goals: maximize revenue, cut costs, win usersMeet CodeClash: LMs compete via their codebases across multi-round tournaments to achieve high-level goals.Because real software dev isn’t about following instructions. It’s about achieving outcomes.Here's how it works:Two LMs enter a tournament. Each maintains its own codebase.Every round:1. Edit Phase: LMs modify their codebases however they like 2. Competition phase: Codebases battle in an arena. 3. RepeatThe LM that wins the majority of rounds is declared winner.Arenas can be anything like games, trading sims, cybersec envs. We currently have 6 arenas implemented and support for 8 different programming languages.This has been one of our biggest projects in terms of scale to date. Over the past few months, we've completed 1.5k tournaments, totalling more than 50,400 agent runs. And you can look at all of these runs right now from your browser (links below!)You can find the rankings on our website (spoiler: Sonnet 4.5 tops the list), but almost more interesting: Humans are still way ahead! In one of our arena, even the worst solution from the human leaderboard is miles ahead of the best LM!And we're not surprised: LMs consistently fail to properly adapt to outcomes, hallucinate about reasons for failure, and produce ever messier codebases with every round.More information:https://codeclash.ai/ https://arxiv.org/pdf/2511.00839 https://github.com/codeclash-ai/codeclash
HN · launched Nov 5, 2025 · 5 upvotes (top 90%) · 0 views
Niche: llm benchmarking tools · 84 launches, 1 in the last 7 days · Explore this niche →
Category and tags
- Horizontal
- Observability & eval
- Developer
- AI-native
- Commercial product
Visit website ↗ Source listing ↗
More like this: all 84 launches in llm benchmarking tools →
Could you build this?
No CodeClash is an advanced AI evaluation platform running multi-round competitive tournaments across complex environments like Halite and BattleSnake with secure code sandboxing and statistical Elo benchmarking. Building and scientifically validating a frontier LLM benchmark requires significant AI research expertise and dedicated execution infrastructure.
What it would actually take: A full version requires containerized, sandboxed multi-agent execution harnesses (Docker/Firecracker) capable of compiling and running arbitrary untrusted code safely at scale. It needs game engine integration, deterministic round-robin or Swiss-tournament matchmaking, dynamic LLM prompt/agent scaffolding, and robust statistical Elo rating algorithms. Building this requires deep ML research expertise in benchmarking methodology, rigorous sandboxing security, and substantial compute budget to run thousands of model iterations.
Discussion
1 comment analyzed.
Feature requests: Integrate reinforcement learning into competitive framework
Competitors
Other products that read as similar to this one : 122 launches clear the similarity bar, closest 8 shown.
Attention rank: #119 of 123 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 4 days after the earliest competitor.
- swe-sweep · github · 2026-10-01 · 55 upvotes · similarity 0.45
- Stage · hn · 2026-04-16 · 130 upvotes · similarity 0.42
- LLMadness · hn · 2026-03-19 · 5 upvotes · similarity 0.42
- Pencil Puzzle Bench · hn · 2026-03-03 · 5 upvotes · similarity 0.40
- Benchmark: AI doesn't find bugs unless you tell it what's wrong · hn · 2026-10-01 · 5 upvotes · similarity 0.40
- AI-Evals.io · hn · 2026-02-15 · 5 upvotes · similarity 0.40
- LLM Skirmish · hn · 2026-02-04 · 5 upvotes · similarity 0.38
- A real-time strategy game that AI agents can play · hn · 2026-02-25 · 220 upvotes · similarity 0.38
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.