genpark-robots-txt-crawl-delay-policy-evaluator-skill
RFC 9309 robots.txt parser with path permissions, Disallow/Allow rules, and crawl delay compliance
Get picks like this daily. Today's launch numbers, the top launches, AI and tech news, and a daily opportunity pick, straight to your inbox.
Want to reach these readers? Sponsor an issue.
GITHUB · launched Sep 29, 2026 · 7 upvotes (top 95%) · 0 views
Niche: data parsing libraries and tools · 71 launches, 1 in the last 7 days · Explore this niche →
Category and tags
- Horizontal
- Dev tools
- Developer
- Not AI
- Hobby / open-source project
- agent-skills
- agentic-browsing
- compliance
- crawl-delay
- developer-tools
- mcp
- +5 more
Visit website ↗ Source listing ↗
More like this: all 71 launches in data parsing libraries and tools →
Could you build this?
Yes RFC 9309 specifies an explicit, deterministic grammar and path matching logic for robots.txt files, which an LLM can easily implement and test against standard test suites.
Competitors
Other products that read as similar to this one : 237 launches clear the similarity bar, closest 8 shown.
Attention rank: #157 of 238 (itself plus its competitors, highest first, normalized so YC and Product Hunt are compared fairly).
Launched 326 days after the earliest competitor.
- Robots.txt Generator · ph · 2026-10-22 · 2 upvotes · similarity 0.52
- Robots.txt Generator · ph · 2026-10-04 · 2 upvotes · similarity 0.51
- genpark-agent-sandboxed-file-diff-safety-auditor-skill · github · 2026-09-26 · 7 upvotes · similarity 0.49
- genpark-agent-tool-permission-rbac-validator-skill · github · 2026-09-26 · 7 upvotes · similarity 0.49
- genpark-covert-data-exfiltration-firewall-skill · github · 2026-09-26 · 7 upvotes · similarity 0.49
- ToolsVale · ph · 2026-09-17 · 1 upvotes · similarity 0.48
- genpark-recursive-descent-ast-parser-skill · github · 2026-09-10 · 7 upvotes · similarity 0.46
- Contomatix LLM SEO Checker · ph · 2026-10-06 · 1 upvotes · similarity 0.46
The niche, labels, vibe code rating and competitor comparison on this page are analysis by Nicheloom. If you use them, please credit nicheloom.com. How you may use our data.