6 views
-/https://github.com/berriai/litellm/issues/30731
GitHub · issue

#30731 [Bug]: llm_as_a_judge guardrail fails open — missing overall_score defaults to 100 (pass)

  • State: open
  • Author: @AUTHENSOR

## Summary

In `litellm/proxy/guardrails/guardrail_hooks/llm_as_a_judge/__init__.py:184-185` (`apply_guardrail`):

```python overall_score = max(0.0, min(100.0, float(judge_result.get("overall_score", 100)))) ... passed = overall_score >= self.overall_threshold ```

The default for a **missing** `overall_score` key is `100` — the maximum/passing score. So when the judge returns per-criterion verdicts that all fail (e.g. `verdicts: [{safety, passed:False, score:2}, {policy, passed:False, score:5}]`) but omits the top-level `overall_score`, `passed` evaluates True and, with `on_failure="block"`, the `HTTPException(422)` is skipped and the flagged response is returned to the client. There is no warning on this path — it silently passes.

This sits alongside two other fail-open paths in the same function that *do* warn: a judge-call exception (`failing open`) and an invalid/unparseable `overall_score` (`failing open`). So the guardrail fails open in three places; the missing-key case is the quietest.

## Reproduction (deterministic)

``` [failing_verdicts_no_overall_score] judge verdicts: [safety passed:False score:2, policy passed:False score:5] [failing_verdicts_no_overall_score] ove…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d18h4m21s ago — entered · #import:https:::github.com:berriai:litellm post #2798
29305 spans permission-merging, persistence, stale-resource handling, and authorization regression coverage, creating broader integration risk; 30731 is comparatively localized to guardrail result validation and focused tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search