Nicheloom visits: 60,536 all-time · 7,913 in the last 7 days

The opportunity tracker for new startups.

Today: 895 launches · Niche of the day: browser-based video games · See what launched today →

New eval from SWE-bench team evalutes LMs based on goals not tickets

Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.

Want to reach these readers? Sponsor an issue.

Current evals test LMs on tasks: "fix this bug," "write a test"But we code to achieve goals: maximize revenue, cut costs, win usersMeet CodeClash: LMs compete via their codebases across multi-round tournaments to achieve high-level goals.Because real software dev isn’t about following instructions. It’s about achieving outcomes.Here's how it works:Two LMs enter a tournament. Each maintains its own codebase.Every round:1. Edit Phase: LMs modify their codebases however they like 2. Competition phase: Codebases battle in an arena. 3. RepeatThe LM that wins the majority of rounds is declared winner.Arenas can be anything like games, trading sims, cybersec envs. We currently have 6 arenas implemented and support for 8 different programming languages.This has been one of our biggest projects in terms of scale to date. Over the past few months, we've completed 1.5k tournaments, totalling more than 50,400 agent runs. And you can look at all of these runs right now from your browser (links below!)You can find the rankings on our website (spoiler: Sonnet 4.5 tops the list), but almost more interesting: Humans are still way ahead! In one of our arena, even the worst solution from the human leaderboard is miles ahead of the best LM!And we're not surprised: LMs consistently fail to properly adapt to outcomes, hallucinate about reasons for failure, and produce ever messier codebases with every round.More information:https://codeclash.ai/ https://arxiv.org/pdf/2511.00839 https://github.com/codeclash-ai/codeclash

HN · launched Nov 5, 2025 · 5 upvotes (top 90%) · 0 views

Niche: llm benchmarking tools · 84 launches, 1 in the last 7 days · Explore this niche →

Category and tags

  • Horizontal
  • Observability & eval
  • Developer
  • AI-native
  • Commercial product

Visit website ↗ Source listing ↗

Could you build this?

No CodeClash is an advanced AI evaluation platform running multi-round competitive tournaments across complex environments like Halite and BattleSnake with secure code sandboxing and statistical Elo benchmarking. Building and scientifically validating a frontier LLM benchmark requires significant AI research expertise and dedicated execution infrastructure.

What it would actually take: A full version requires containerized, sandboxed multi-agent execution harnesses (Docker/Firecracker) capable of compiling and running arbitrary untrusted code safely at scale. It needs game engine integration, deterministic round-robin or Swiss-tournament matchmaking, dynamic LLM prompt/agent scaffolding, and robust statistical Elo rating algorithms. Building this requires deep ML research expertise in benchmarking methodology, rigorous sandboxing security, and substantial compute budget to run thousands of model iterations.

Discussion

1 comment analyzed.

Feature requests: Integrate reinforcement learning into competitive framework

Competitors

Other products that read as similar to this one : 122 launches clear the similarity bar, closest 8 shown.

Attention rank: #119 of 123 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).

Launched 4 days after the earliest competitor.

The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.