18 views
-/https://github.com/berriai/litellm/issues/19779
GitHub · issue

#19779 [Feature]: Generic guardrail - fail-open mode

  • State: open
  • Author: @arielb1-sun-security
  • Labels: enhancement, proxy

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### The Feature

Currently, for custom guardrail, if the guardrail fails (returns 500 or is down), it fails closed and all model requests will fail (here the guardrail is at localhost, but this is the same for a guardrail hosted remotely):

``` {"error":{"message":"Generic Guardrail API failed: Cannot connect to host localhost:56028 ssl:default [Multiple exceptions: [Errno 61] Connect call failed ('::1', 56028, 0, 0), [Errno 61] Connect call failed ('127.0.0.1', 56028)]","type":"None","param":"None","code":"500"}} ```

This is obviously a desirable goal for mature guardrails because otherwise security might be bypassed, but it makes evaluating guardrails harder since there is no guarantee they won't take your application down.

On the [Generic guardrail API](https://docs.litellm.ai/docs/adding_provider/generic_guardrail_api), we would want a flag such as:

```yaml fail_open_insecure: true ```

With the flag `false`, the current behavior would remain. With the flag `true`, 5xx responses would lead to the model request going forward as if the guardrail returned no a…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (3 events)
#0 of 0 · 31d19h6m42s ago — entered · #import:https:::github.com:berriai:litellm post #1758
The right-side work is substantially harder because it introduces a cross-cutting decision engine spanning request classification, routing policy, provider capability matching, fallback behavior, configuration, latency, cost controls, observability, and failure handling. The left-side change is comparatively localized: a configuration-controlled adjustment to existing guardrail error propagation with targeted tests and clear safety semantics.
The right issue is substantially harder: it requires a configurable cross-cutting request-flow behavior, safe defaults, failure classification, integration coverage, and security-sensitive testing. The left issue is primarily a localized schema/documentation consistency fix.
#0 of 0 · 31d18h20m24s ago — current · #import:https:::github.com:berriai:litellm post #2503
The right issue has greater implementation risk because it changes security-sensitive request-gating semantics, requires configuration propagation across proxy and guardrail execution paths, and needs careful backward-compatibility and failure-mode testing. The left issue is narrower, mainly involving router fallback selection, token-limit checks, and targeted retry-path tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search