Langfuse AI Gateway, open source rust gateway to trace every LLM call

Hi HN, this is Max, Marc and Clemens, co-founders of Langfuse, the open source tool for agent & observability & evals (https://github.com/langfuse/langfuse).It’s been a minute since our last Show HN in August 2023 (https://news.ycombinator.com/item?id=37310070) - we’ve been busy building out and serving our tracing & eval pieces over the past years and have been very lucky to build a successful company and open source project around it. Today, we are happy to show the beta of Langfuse AI Gateway: https://github.com/langfuse/langfuse/tree/main/ai-gateway. You may ask, why an AI Gateway as our second product: it’s been one our community’s strongest asks + integration pattern that we’ve seen. We want to offer a scalable, reliable and secure way to solve for the gateway use case in the same way that we’ve done with tracing/evals over the past years. And we’ve recently grown our team with some excellent folks - expect to see more bigger bets from us.Our observability SDKs currently see the code you instrument. Much of a company's model traffic runs through code that is not or cannot be easily instrumented: coding agents like Pi, Claude Code and Codex, internal tools, scripts etc. Third party gateways are already popular ways to emit traces to Langfuse and we’re happy to now ship one natively.You change your base URL and your API key. Langfuse AI Gateway forwards the native request with a provider key that does not need to be distributed across the team. Right now, it doesn't translate between APIs, responses are relayed unchanged. This was a conscious decision to ensure stability for gateway users, as translation between e.g. OpenAI responses and Anthropic messages are imperfect and might lead to subtle failures in your agents.If you pass a traceparent header, the call lands inside your existing Langfuse traces.You can try by docker composing langfuse up and then on to http://localhost:3000--git clone https://github.com/langfuse/langfuse.gitcd langfusedocker compose up--On Cloud, each call becomes a Langfuse generation with tokens, cost, latency and time to first token. Prompts and outputs are recorded if you opt in. If you self-hosted you can also track nothing, not even metadata.Some details:- Written in Rust. Stateless, with no database; it talks only to the Langfuse platform and the providers.- 22MB Docker image. Under 3ms median overhead with tracing on, 1,000+ concurrent streams on 2 vCPUs.- MIT licensed. Self-host it with Docker Compose or Helm or use our Cloud version.- Your provider keys and contracts. We don't resell tokens.What’s next? - Budget caps per key, user, team- Custom model aliases and fallbacks- Guardrails- PII redaction- SCIM user provisioningWe’ll be around & looking forward to all questions & comments