6 views
-/https://github.com/berriai/litellm/issues/31870
GitHub · issue

#31870 bug(akto-guardrail): during_call mode silently no-ops; missing model-group scoping and O(n^2) streaming ingest

  • State: open
  • Author: @deepanshululla

## Summary

The Akto guardrail integration has three issues in OSS 1.91.0.

First, setting \`mode: during_call\` in the guardrails YAML produces no validation effect and no error. \`HOOK_TO_INPUT\` at line 44 of \`akto.py\` only maps \`pre_call\` and \`post_call\`; \`during_call\` is absent. Additionally, \`supported_event_hooks\` only lists \`pre_call\` and \`post_call\`, so the framework skips the hook entirely before \`apply_guardrail\` is ever called. Any deployment configured with \`mode: during_call\` receives zero guardrail coverage.

Second, there is no way to scope Akto invocations to specific model groups. Operators who want to run Akto only for some model groups (e.g., by tier or cost) cannot do so without patching the integration.

Third, the streaming post_call path calls the Akto ingest endpoint once per every 5th chunk, serializing the growing \`responses_so_far\` buffer each time. For a 500-chunk stream this is 100 Akto calls, each serializing a longer buffer, producing O(n^2) work that stalls the asyncio event loop. Additionally, two hot-path log statements eagerly format large response objects even when DEBUG logging is disabled.

## Reproduce

```python from lite…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (4 events)
#0 of 0 · 31d18h47m40s ago — entered · #import:https:::github.com:berriai:litellm post #2169
The left requires coordinated changes across runtime behavior, configuration, and performance-sensitive paths, creating broader regression risk and testing scope. The right is a contained dependency and lockfile maintenance change.
Issue 31870 spans multiple integration layers, lifecycle hooks, configuration scoping, streaming performance, and regression testing, creating substantially broader implementation and validation risk than the focused authorization-path fix in issue 34975.
31870 requires coordinated changes across guardrail lifecycle wiring, configuration scoping, streaming-path performance, and regression coverage, whereas 18060 is primarily a focused retry/error-classification investigation. The broader cross-cutting scope and concurrency-sensitive behavior make 31870 riskier and harder.
#0 of 0 · 31d18h13m37s ago — current · #import:https:::github.com:berriai:litellm post #2718
Issue 31870 requires coordinated framework, configuration, and performance changes with broader compatibility and asynchronous regression testing; issue 35229 is a localized provider-response mapping fix with a narrower regression surface.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search