9 views
-/https://github.com/berriai/litellm/issues/34922
GitHub · issue

#34922 [Bug]: Noma v2 guardrail ignores streaming_end_of_stream_only and streaming_sampling_rate

  • State: open
  • Author: @yryzhan-vitech
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

The Noma Security v2 guardrail (noma_v2, also reachable as guardrail: noma with use_v2: true) does not honor the shared streaming knobs streaming_end_of_stream_only and streaming_sampling_rate. Setting either in config.yaml has no effect: streaming post-call scans always run per chunk at the hard-coded default sampling rate of 5. There is no error and no warning, so the misconfiguration is invisible.

NomaV2Guardrail implements apply_guardrail and does not define its own async_post_call_streaming_iterator_hook. litellm/proxy/utils.py therefore classifies it as an apply_guardrail callback and routes its streaming post-call through UnifiedLLMGuardrails.async_post_call_streaming_iterator_hook:

litellm/proxy/utils.py:

if "async_post_call_streaming_iterator_hook" in cls_attrs: has_iterator_override = True iterator_overrides.append((resolved, "override")) elif "apply_guardrail" in cls_attrs: iterator_overrides.append((resolved, "apply_guardrail"))

That dispatcher resolves the knobs off the guardrail instance with getatt…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h41m2s ago — entered · #import:https:::github.com:berriai:litellm post #3178
The right issue is harder because it spans container-image layout, filesystem permissions, migration tooling, and Kubernetes/Helm job behavior, whereas the left issue is primarily a localized guardrail-dispatch and configuration-propagation fix with targeted tests.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search