Optimizing LiteLLM with Rust

Details

External ID

45968461

Source

HN

Company

—

Product

Optimizing LiteLLM with Rust

Website domain

github.com

Launched

Nov. 18, 2025

Cohort

—

Upvotes

27

Upvotes percentile

0.7390829694323144

Tags

—

Fetched at

Sept. 7, 2026, 9:25 p.m.

Updated at

Sept. 7, 2026, 9:25 p.m.

Description

I've been working on Fast LiteLLM - a Rust acceleration layer for the popular LiteLLM library - and I had some interesting learnings that might resonate with other developers trying to squeeze performance out of existing systems.My assumption was that LiteLLM, being a Python library, would have plenty of low-hanging fruit for optimization. I set out to create a Rust layer using PyO3 to accelerate the performance-critical parts: token counting, routing, rate limiting, and connection pooling.The Approach- Built Rust implementations for token counting using tiktoken-rs- Added lock-free data structures with DashMap for concurrent operations- Implemented async-friendly rate limiting- Created monkeypatch shims to replace Python functions transparently- Added comprehensive feature flags for safe, gradual rollouts- Developed performance monitoring to track improvements in real-timeAfter building out all the Rust acceleration, I ran my comprehensive benchmark comparing baseline LiteLLM vs. the shimmed version:Function Baseline Time Shimmed Time Speedup Improvement Statustoken_counter 0.000035s 0.000036s 0.99x -0.6%count_tokens_batch 0.000001s 0.000001s 1.10x +9.1%router 0.001309s 0.001299s 1.01x +0.7%rate_limiter 0.000000s 0.000000s 1.85x +45.9%connection_pool 0.000000s 0.000000s 1.63x +38.7%Turns out LiteLLM is already quite well-optimized! The core token counting was essentially unchanged (0.6% slower, likely within measurement noise), and the most significant gains came from the more complex operations like rate limiting and connection pooling where Rust's concurrent primitives made a real difference.Key Takeaways1. Don't assume existing libraries are under-optimized - The maintainers likely know their domain well 2. Focus on algorithmic improvements over reimplementation - Sometimes a better approach beats a faster language 3. Micro-benchmarks can be misleading - Real-world performance impact varies significantly 4. The most gains often come from the complex parts, not the simple operations 5. Even "modest" improvements can matter at scale - 45% improvements in rate limiting are meaningful for high-throughput applicationsWhile the core token counting saw minimal improvement, the rate limiting and connection pooling gains still provide value for high-volume use cases. The infrastructure I built (feature flags, performance monitoring, safe fallbacks) creates a solid foundation for future optimizations.The project continues as Fast LiteLLM on GitHub for anyone interested in the Rust-Python integration patterns, even if the performance gains were humbling.Edit: To clarify - the negative performance for token_counter is likely in the noise range of measurement, suggesting that LiteLLM's token counting is already well-optimized. The 45%+ gains in rate limiting and connection pooling still provide value for high-throughput applications.

• Source listing
• Website