6 views
-/https://github.com/berriai/litellm/issues/32528
GitHub · issue

#32528 [Bug]: LiteLLM Semantic Tool Search creates embeddings without batching, causing hangs and failures

  • State: open
  • Author: @nguyenmp
  • Labels: bug, proxy, llm translation

### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

I have [MCP Semantic Tool Filters](https://docs.litellm.ai/docs/mcp_semantic_filter) enabled and it's causing my instance to hang often trying to list tools. It seems to be trying to generate the embeddings for all the tools I have, which is greater than the input limit of the API.

It'd be nice if LiteLLM batched calls to embedding models when they're too big, and if I could configure that in the model config.

Alternatively, specifying a batch size fro MCP Semantic Tool Filtering would also work but would only fix this for this isolated use-case I ran into.

https://github.com/BerriAI/litellm/blob/litellm_internal_staging/litellm/llms/openai_like/embedding/handler.py => here the code passes input into

As an aside, any of these above fixes would also probably help with https://github.com/BerriAI/litellm/issues/26155

### Steps to Reproduce

1. Add > 2048 tools to MCP 2. Enable MCP Semantic Search on DeepInfra (limit 1024) or OpenAI (limit 2048) 3. See logs 4. Notably, tool-calls take a long time to list out things because the index isn't po…

GitHub resolver

Import GitHub neighbors on demand. Results are saved as system ingests.

Refresh page
vote history (1 events)
#0 of 0 · 31d17h52m27s ago — entered · #import:https:::github.com:berriai:litellm post #3096
The left issue requires designing reusable batching, size-limit handling, configuration, retries, and integration across embedding and MCP execution paths, with concurrency and regression risks. The right issue is primarily a localized identifier-mapping and persistence correction with narrower compatibility concerns.
discussed in #import:https:::github.com:berriai:litellm

ranked child groups

no voted pairs yet in this scope

cli
src
spread
search