20 views
-/https://github.com/berriai/litellm/issues/25255
GitHub · issue

#25255 [Feature Request]: Support for NVIDIA NeMo Guardrails (Python Library & Proxy API Server)

  • State: open
  • Author: @metalshanked
  • Labels: llm translation

**Is your feature request related to a problem? Please describe.** As enterprise adoption of LLMs grows, so does the need for robust, programmable guardrails to ensure output safety, prevent hallucinations, and restrict topic deviation. While LiteLLM currently supports some guardrail providers (like Lakera, Prompt Injection, and Presidio), it lacks native support for **NVIDIA NeMo Guardrails**, which has become an industry standard for open-source, highly customizable guardrails.

Currently, if we want to use NeMo Guardrails with LiteLLM, we have to run NeMo Guardrails as a separate upstream service or wrap LiteLLM calls in custom code. This adds architectural complexity, increases latency, and prevents us from utilizing LiteLLM's native Proxy Server features (like spend tracking, load balancing, and API key management) as the primary entry point.

**Describe the solution you'd like** I would like to see native integration of NVIDIA NeMo Guardrails into LiteLLM, specifically within the **Proxy API Server** and the standard Python library.

Ideally, it would function similarly to existing LiteLLM guardrail integrations, where we can specify a path to the NeMo Guardrails configuratio…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (9 events)
#0 of 0 · 31d18h33m15s ago — entered · #import:https:::github.com:berriai:litellm post #2302
The NeMo integration spans multiple public layers, provider behavior, configuration, request lifecycles, and compatibility testing, creating substantially broader implementation and maintenance risk. The router issue is a focused authorization/data-propagation fix with targeted regression coverage.
Native NeMo Guardrails support spans multiple public interfaces, configuration and dependency management, request/response interception, and compatibility with existing proxy guardrail flows. The Redis issue is high-risk but more narrowly scoped to diagnosing request isolation, cache keying, and distributed concurrency behavior.
Native NeMo Guardrails support spans new provider integration, configuration and lifecycle handling, proxy/API behavior, dependency compatibility, and broad testing. The MCP failure is narrower in scope, though asynchronous cancellation and an upstream SDK interaction create meaningful debugging risk.
The right issue is substantially harder because it requires a new cross-cutting guardrails integration spanning external dependency management, configuration, request/response interception, proxy lifecycle, Python APIs, compatibility, and end-to-end testing. The left issue is comparatively contained adapter debugging across a few translation paths, with focused regression coverage.
29320 is harder due to cross-cutting request-path behavior, configuration inheritance, fail-open guarantees, dependency/runtime concerns, and coordinated SDK, proxy, observability, and testing work. 25255 mainly requires adapting an external guardrails framework to established integration and proxy extension points, though its lifecycle and compatibility risks remain significant.
The left item is harder because it spans multiple public interfaces, introduces a complex external runtime dependency, and requires careful handling of configuration, request flows, streaming, errors, deployment, and compatibility testing. The right item is narrower in scope: primarily persistence-layer recalculation, policy evaluation, an administrative endpoint, and auditability concerns.
28168 spans cross-resource state discovery, normalization, secret handling, dependency ordering, and reliable import/export semantics, creating substantially broader migration and correctness risk. 25255 is a focused provider integration with middleware and configuration work, but can follow existing guardrail extension patterns.
The left task has substantially broader scope: it requires designing and maintaining a new external-provider integration across both library and proxy layers, including configuration, lifecycle behavior, compatibility, and comprehensive testing. The right task is a localized proxy configuration/loading defect with a narrower diagnostic and remediation path.
#0 of 0 · 31d17h19m18s ago — current · #import:https:::github.com:berriai:litellm post #3531
The left item requires coordinated changes across multiple request paths, stateful streaming behavior, accounting, and compatibility testing, creating substantially greater regression risk. The right item is primarily a bounded third-party integration with configuration and lifecycle work.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search