### Check for existing issues

- [x] I have searched the existing issues and checked that my issue is not a duplicate.

### The Feature

### 🚀 Feature Request: Native Bayesian Optimization for Hyperparameter & Prompt Tuning

#### Is your feature request related to a problem? Please describe. Currently, finding the optimal inference hyperparameters (`temperature`, `top_p`, `frequency_penalty`) or system prompt structures in LiteLLM is a manual trial-and-error process. While LiteLLM is excellent at routing and tracking costs/latency, it lacks a native automated optimization loop. Setting up external frameworks like Optuna or W&B Sweeps requires substantial boilerplate code to feed LiteLLM responses into an evaluation metric and loop back.

#### Describe the solution you'd like We would like to introduce a native or pluggable **Bayesian Optimization / Automated Tuning module** into LiteLLM (both Python SDK and LiteLLM Proxy). This feature would allow users to define a hyperparameter search space and an evaluation metric directly in LiteLLM to automatically find the highest-performing configuration (balancing accuracy, cost, and latency).

#### Key Components Proposed:

1. **`litellm.t…