Nicheloom visits: 62,674 all-time · 8,415 in the last 7 days

The opportunity tracker for new startups.

Today: 1,441 launches · Niche of the day: digital marketing agency services · See what launched today →

TrnRT

An open-source LLM inference engine for AWS Trainium

Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.

Want to reach these readers? Sponsor an issue.

Hi HN, Sanchit here from RunAnywhere (YC W26). We just open-sourced TrnRT: Trainium RunTime, our inference engine for AWS Trainium. We started 3 weeks back, and currently we’re able to serve and run GLM 5.3 flash (FP8) on AWS trn2.48xlarge, and will bring more models support soon. It’s open to contributions :)What it does today:- Serves 2 instances of GLM-5.3 flash on a single box, 8 trn2 with a cache aware router.- Can process 3.57M TPM (tokens per minute) on SemiAnalysis's AgentX benchmarks.- Runs decode as one NKI kernel per layer half, or for the 11 sparse-attention layers fuses both blocks into a single kernel, for faster performance.We started working on Trn2 because we wanted to explore it and see if we can come close to beating it’s performance in terms of cost efficiency compared to H100/H200. A big MoE with 1M context window such as GLM-5.3 flash requires a lot of memory, and memory/tb on a trn2.48xlarge costs less than half compared to 8x h200 on AWS, and hence we wanted to experiment with it and now we’re sharing some early results.But AWS's own stack didn't support the model: vLLM Neuron's latest release did not register any GLM model, and NxDI, AWS's own distributed inference framework, is no longer under active development. We also saw the drop of SGLang Neuron implementation yesterday, but there is no good support for Trn2, which is the only chip available for public use right now.We ran semiAnalysis's AgentX benchmarks, which replays real coding agent sessions:- 1 box served 28.9 RPM(requests per minute) with 32 agents at batch 12 (also known as concurrency)-The P50 TTFT with a cached prompt at 16 agents is ~1.24 seconds with a decode of 35.9 tps.Initially, the recurrence in one kda (linear attention) layer was ~2% of the layer's attention time at batch 1, and the rest was projections, transposes and XLA glue between small kernels.So we wrote the megakernel so that it covers the whole layer in 2 kernel dispatch, which made decode 25% faster at batch 8.The sparse attention kernel keep a layer’s working weights in SBUF, distributing weighted streaming across the cores and handling the interchip communication withinThere’s an approximation in the sparse-attention prefill, affecting about 1.2% of block selections.You can refer to the write-up for the profiling results:https://runanywhere.ai/blog/glm-5-3-flash-trainium2-nki-mega...Please give it a try yourself on a trn2.48xlarge and reproduce the AgentX results by following the steps in docs/RESULTS.md. Happy to collaborate and hear any feedback.Deploying high performant dedicated inference at lab like performance and half the price for open models such as the GLM-5.3 family, Kimi K3, Deepseek flash etc. is how we make money. You can talk to us regarding the same at [email protected] hope you find this useful. If you are working with Trainium or have experience with NKI kernels, we'd love to get your honest feedback. Feel free to share the models you want support for next, and we welcome any contributions to the repository.

HN · launched Nov 17, 2026 · 46 upvotes (top 18%) · 0 views

Niche: llm inference engines · 137 launches, 5 in the last 7 days · Explore this niche →

Category and tags

  • Horizontal
  • Model & infra
  • Developer
  • AI-native
  • Hobby / open-source project

Visit website ↗ Source listing ↗

Could you build this?

No An LLM inference engine for AWS Trainium requires low-level Neuron SDK kernels, model-parallel sharding, and hardware-specific performance tuning, which is specialized systems work.

What it would actually take: It needs deep knowledge of the Neuron compiler and runtime, custom kernels for attention and FP8 matmul, KV-cache management, and tensor-parallel scheduling across trn2 chips. Building it requires ML systems engineers with accelerator experience and access to Trainium hardware.

Discussion

2 comments analyzed.

Competitors mentioned: sglang

Concerns raised: tokens per minute is an unusual metric

Feature requests: report tokens per second

Competitors

Other products that read as similar to this one : 850 launches clear the similarity bar, closest 8 shown.

Attention rank: #198 of 851 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).

Launched 381 days after the earliest competitor.

The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.