ACE
A dynamic benchmark measuring the cost to break AI agents
Details
- External ID
- 47654123
- Source
- HN
- Company
- —
- Product
- ACE
- Website domain
- fabraix.com
- Launched
- April 5, 2026
- Cohort
- —
- Upvotes
- 9
- Upvotes percentile
- 0.5501285347043702
- Tags
- —
- Fetched at
- Sept. 7, 2026, 9:26 p.m.
- Updated at
- Sept. 7, 2026, 9:26 p.m.
Description
We built Adversarial Cost to Exploit (ACE), a benchmark that measures the token expenditure an autonomous adversary must invest to breach an LLM agent. Instead of binary pass/fail, ACE quantifies adversarial effort in dollars, enabling game-theoretic analysis of when an attack is economically rational.We tested six budget-tier models (Gemini Flash-Lite, DeepSeek v3.2, Mistral Small 4, Grok 4.1 Fast, GPT-5.4 Nano, Claude Haiku 4.5) with identical agent configs and an autonomous red-teaming attacker.Haiku 4.5 was an order of magnitude harder to break than every other model; $10.21 mean adversarial cost versus $1.15 for the next most resistant (GPT-5.4 Nano). The remaining four all fell below $1.This is early work and we know the methodology is still going to evolve. We would love nothing more than feedback from the community as we iterate on this.
Enrichment
- Theme
- algorithmic trading bots and platforms
- Vertical
- Security
- Function
- Observability & eval
- Audience
- Developer
- AI stance
- AI feature
- Project type
- Hobby / open-source project
- Normalized one-liner
- benchmark for measuring cost to break ai agents
- Manually corrected
- False
Could you build this?
Partial Running basic jailbreak prompts against an API is easy, but developing an automated, dynamic multi-agent red-teaming benchmark that simulates economic adversaries and quantifies dollar costs requires novel security research.
What it would actually take: The system requires an adversarial agent framework (e.g., using LangGraph or AutoGen) paired with target agent environments (sandboxed tool environments like mock banking or terminal access). The adversarial agent uses reinforcement learning or tree-search jailbreaking strategies to autonomously explore attack vectors while tracking exact token/compute consumption per target API call. Validating reliable exploit criteria and modeling game-theoretic cost curves demands specialized AI security, red-teaming, and vulnerability research expertise.
Discussion
3 comments analyzed.
Competitors mentioned: Other major LLM models (tested in jailbreak studies)
Competitors
Other products that read as similar to this one — 557 launches clear the similarity bar, closest 8 shown.
Attention rank: #230 of 558 (itself plus its competitors, highest first — normalized so YC and Product Hunt are compared fairly).
Launched 156 days after the earliest competitor.
- Simreal-MLBench · github · 2026-09-21 · 71 upvotes · similarity 0.48
- ReasonBlocks - Stop your AI agents from burning money re-learning what they already know. · yc · 2026-04-27 · 23 upvotes · similarity 0.47
- We post-trained a model that pen tests instead of refusing · hn · 2026-06-20 · 93 upvotes · similarity 0.47
- Token Economics Calculator for AI inference hardware · hn · 2025-11-19 · 13 upvotes · similarity 0.45
- genpark-adversarial-prompt-jailbreak-detector-skill · github · 2026-09-26 · 7 upvotes · similarity 0.45
- genpark-minimax-alpha-beta-adversarial-search-skill · github · 2026-09-09 · 8 upvotes · similarity 0.45
- genpark-minimax-alpha-beta-adversarial-search-skill · github · 2026-09-09 · 8 upvotes · similarity 0.45
- Open Benchmarks Grants– a $3M commitment to close the AI eval gap · hn · 2026-02-11 · 6 upvotes · similarity 0.45
Other launches for this product
- No other launches for this product.
Same idea, different domain
Nobody's really built a observability & eval tool for Media & entertainment yet.