20 views
-/https://github.com/berriai/litellm/issues/31263
GitHub · issue

#31263 LiteLLM Rust Migration - the fastest and litest AI Gateway (sub 1ms overheads)

  • State: open
  • Author: @ishaan-berri

Starting this issue as the parent ticket for our rust migration, please post questions or issues here related to the rust migration

Blog Post: https://docs.litellm.ai/blog/litellm-rust-launch

Sign up for early Beta Tester Group: [here](https://docs.google.com/forms/d/e/1FAIpQLSecWdOjkzjEson2UiZpDftOoZPs8RQbtlAM40KSvDXZqEgYaA/viewform)

_TLDR: Over the past year, we have heard the same thing from our users and our community: they want the fastest, most lightweight AI gateway they can run. We have heard you. We are addressing it by moving LiteLLM to Rust, and committing to sub-1ms overhead with a sub-100MB memory binary you can deploy. By the end of this migration, you will get a pure Rust server that can serve 100% of your AI traffic, with every hot path operation, including auth and rate limiting, running in Rust._

**Note:** FYI - if users want new features / models / providers on AI Gateway I'm happy to add it for them

Timeline

Aug 15, 2026 | litellm.ocr() for Mistral, then all of litellm.ocr(), then the /ocr route -- | -- Sep 1, 2026 | Same pattern for /messages, then /chat/completions Sep 15, 2026 | The router: load balancing, fallbacks, retries, cooldowns Dec 1, 2026 | Th…

Open on GitHub

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (12 events)
#0 of 0 · 31d17h55m27s ago — entered · #import:https:::github.com:berriai:litellm post #2945
-/https://github.com/berriai/litellm/issues/31263⭐20:1-/https://github.com/berriai/litellm/issues/25628
The left task is a repository-wide cross-language migration involving architecture, feature parity, performance validation, deployment, and substantial regression risk. The right task is a narrowly scoped streaming callback mapping bug with localized code changes and targeted tests.
#0 of 0 · 31d17h37m58s ago · #import:https:::github.com:berriai:litellm post #3252
-/https://github.com/berriai/litellm/issues/31263⭐20:1-/https://github.com/berriai/litellm/issues/31821
The right issue is substantially harder because it entails a broad, cross-cutting systems migration with performance, compatibility, and operational risks. The left issue is a bounded proxy feature involving usage accounting, persistence, APIs, and UI integration.
#0 of 0 · 31d17h37m55s ago · #import:https:::github.com:berriai:litellm post #3253
-/https://github.com/berriai/litellm/issues/31263⭐10:1-/https://github.com/berriai/litellm/issues/30501
#31263 entails a broad cross-language gateway rewrite with compatibility, performance, deployment, and staged-migration risks, while #30501 is a comparatively localized provider-specific request transformation.
#0 of 0 · 31d17h37m51s ago · #import:https:::github.com:berriai:litellm post #3254
-/https://github.com/berriai/litellm/issues/31263⭐10:1-/https://github.com/berriai/litellm/issues/26897
The left issue requires a broad, high-risk rewrite of the gateway across languages, performance-critical paths, compatibility layers, and deployment architecture. The right issue is a narrowly scoped integration bug likely addressed through targeted parameter propagation and Azure request-building fixes.
#0 of 0 · 31d17h37m48s ago · #import:https:::github.com:berriai:litellm post #3255
-/https://github.com/berriai/litellm/issues/31263⭐20:1-/https://github.com/berriai/litellm/issues/30460
A cross-language gateway rewrite spanning core execution paths, compatibility, performance, deployment, and staged migration carries far greater engineering scope and regression risk than a focused distributed-state accounting fix.
#0 of 0 · 31d17h37m38s ago · #import:https:::github.com:berriai:litellm post #3258
-/https://github.com/berriai/litellm/issues/31263⭐20:1-/https://github.com/berriai/litellm/issues/31568
The Rust migration is a sweeping, cross-cutting architectural rewrite involving compatibility, performance, security-sensitive gateway paths, rollout coordination, and extensive regression testing. The provider addition is comparatively contained, primarily requiring API translation, model metadata, pricing, and proxy integration.
#0 of 0 · 31d17h37m31s ago · #import:https:::github.com:berriai:litellm post #3260
-/https://github.com/berriai/litellm/issues/31263⭐20:1-/https://github.com/berriai/litellm/issues/32456
The left task is a broad, high-risk systems migration spanning architecture, language boundaries, performance targets, compatibility, and many subsystems. The right task is comparatively localized to one provider integration and request-translation path, with narrower testing and deployment impact.
#0 of 0 · 31d17h37m1s ago · #import:https:::github.com:berriai:litellm post #3268
-/https://github.com/berriai/litellm/issues/31263⭐6:1-/https://github.com/berriai/litellm/issues/34241
The left issue is substantially harder: it entails a large-scale runtime and language migration across core functionality, with compatibility, performance, rollout, and regression risks. The right issue is broad but comparatively more bounded to auditing ownership boundaries, reorganizing code, and documenting or enforcing licensing distinctions.
#0 of 0 · 31d17h36m19s ago · #import:https:::github.com:berriai:litellm post #3280
-/https://github.com/berriai/litellm/issues/31263⭐20:1-/https://github.com/berriai/litellm/issues/21347
The left issue is substantially harder: it entails a broad cross-language platform migration, preserving behavior and compatibility across many execution paths while meeting stringent performance and deployment goals. The right issue is primarily a bounded standards-validation and test-infrastructure effort, with narrower implementation and rollout risk.
#0 of 0 · 31d17h34m50s ago · #import:https:::github.com:berriai:litellm post #3305
-/https://github.com/berriai/litellm/issues/31263⭐20:1-/https://github.com/berriai/litellm/issues/29320
Issue #31263 is substantially harder because it is a broad, multi-phase systems migration involving a new implementation language, parity across gateway hot paths, performance and memory targets, compatibility, deployment, and long-term rollout risk. Issue #29320 is a bounded opt-in integration with localized callback, configuration, observability, and fallback behavior.
#0 of 0 · 31d17h22m33s ago · #import:https:::github.com:berriai:litellm post #3507
-/https://github.com/berriai/litellm/issues/31263⭐10:3-/https://github.com/berriai/litellm/issues/31595
The Rust migration is substantially harder because it is a large architectural rewrite requiring broad behavioral parity, performance validation, interoperability, and long-term maintenance across the gateway. The ADEPT work is a complex but bounded feature spanning routing, persistence, training integration, and dashboard operations, with fewer system-wide compatibility risks.
#0 of 0 · 31d17h20m53s ago — current · #import:https:::github.com:berriai:litellm post #3528
-/https://github.com/berriai/litellm/issues/31263⭐10:3-/https://github.com/berriai/litellm/issues/35109
The left requires a broad cross-language platform rewrite with extensive compatibility, performance validation, and staged migration risk, while the right is a narrower proxy and storage-isolation feature with bounded integration work.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search