### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### What happened?

A completion fails with ``` Message: litellm.MidStreamFallbackError: litellm.RateLimitError: litellm.RateLimitError: vertex_ai_betaException - b'{\n "error": {\n "code": 429,\n "message": "Resource exhausted. Please try again later. Please refer to https://cloud.google.com/vertex-ai/generative-ai/docs/error-code-429 for more details.",\n "status": "RESOURCE_EXHAUSTED"\n }\n}\n'No fallback model group found for original model_group=gemini-flash. Fallbacks=[]. Received Model Group=gemini-flash Available Model Group F... (litellm_truncated skipped 69 chars. Truncation is a DB storage safeguard. [..] ```

And it says in litellm UI: ``` Retries: None ```

However, in my litellm config I have: ``` router_settings: num_retries: 3 # retry each model group before falling back retry_after: 2 # seconds between retries ```

So I don't understand why it doesn't seem to do a retry?

### Steps to Reproduce

1. Use vertex gemini flash 3.5 2. Use it and wait for a 429 to appear 3. Look in the LitellmUI logs

### Relevant log output

```s…