Nicheloom visits: 62,190 all-time · 8,336 in the last 7 days

The opportunity tracker for new startups.

Today: 2,085 launches · Niche of the day: browser-based utility tools · See what launched today →

Autonomous recovery for distributed training jobs

Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.

Want to reach these readers? Sponsor an issue.

Hi HN! We’re TensorPool. We help companies access and optimize large scale compute for training foundation models.The ProblemIt’s been almost a year since we’ve finished YC, and we’ve just crossed 100,000 multinode training GPU hours run on our platform.On those training runs, we’ve seen countless 3am job crashes because of issues like an Xid error from a flaky GPU or an S3 timeout that corrupted a checkpoint save. By the time you wake up and notice, you've lost 8+ hours of compute. You scramble to diagnose the issue, manually restart from the last checkpoint, and hope it doesn't happen again. Rinse and repeat.For training runs that take days to weeks, this constant babysitting is exhausting and expensive. The research iteration cycles lost can also make or break a model release (especially for short reservations).What We BuiltThis agent monitors your training jobs and autonomously recovers them when things go wrong. It works with Kubernetes, Slurm, and TensorPool Jobs.We originally built the TensorPool Agent as an internal tool to help us debug failures with our own customers. Over time, we realized its performance was so good that we could automate the entire triage process. We're now releasing a public beta for people to use.Best case: The TensorPool Agent detects the failure, diagnoses the root cause, fixes it, and restarts your job from the last checkpoint – all while you sleep ;)Worst case: If the TensorPool agent can't fix the issue automatically, it delivers a preliminary RCA and a list of actions it attempted, giving you a head start on debugging.How It Works1) Registration – You provide credentials to your job scheduler via our dashboard. Perms are granted on a whitelist basis; you explicitly control what actions the agent can take.2) Monitoring – The agent continuously monitors your job for failure conditions.3) Recovery – On failure, the agent analyzes logs and attempts to diagnose the issue. If successful, it restarts the job from the last checkpoint and resumes monitoring. If not, you get an alert with full context.Target Failure ModesThe agent is specifically designed for runtime errors that occur deep into training, like:- CUDA OOM: Memory leaks, gradient explosions- Xid errors: GPU hardware faults (Xid 79, 63, 48, etc.)- Distributed communication failures: NCCL timeouts, rank failures- Storage I/O errors: Checkpoint corruption- Network issues: S3 request timeouts on mounted object storage

HN · launched Jan 29, 2026 · 12 upvotes (top 44%) · 0 views

Niche: gpu infrastructure and developer tools · 132 launches, 6 in the last 7 days ▼ -50% · Explore this niche →

Category and tags

  • Horizontal
  • Model & infra
  • Developer
  • AI-native
  • Commercial product

Visit website ↗ Source listing ↗

Could you build this?

No Autonomous recovery for multi-node distributed AI training requires deep systems programming, NCCL/InfiniBand network diagnostics, and GPU hardware cluster orchestration at scale.

What it would actually take: Requires an agent monitoring low-level GPU metrics, NVLink/NCCL error logs, and InfiniBand status across hundreds of bare-metal nodes. The system must integrate directly with distributed frameworks (PyTorch DDP, Megatron-LM) to execute rapid zero-bubble elastic checkpointing and live node-swapping without killing entire jobs. This requires deep systems engineering, kernel-level debugging, and access to expensive multi-node GPU clusters.

Discussion

3 comments analyzed.

Concerns raised: Silent failures hard to detect (NCCL hangs, gradient explosions without crashes), Relies on explicit error logs rather than behavioral signals, Difficulty distinguishing job running normally vs. running but degraded

Feature requests: Automatic detection of silent failures and hung processes, Business metrics-based alerting instead of just raw measurements, Recovery/restart mechanisms for long-running training jobs

Competitors

Other products that read as similar to this one : 362 launches clear the similarity bar, closest 8 shown.

Attention rank: #183 of 363 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).

Launched 92 days after the earliest competitor.

The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.