LiteLLM

LiteLLM is a tool from BerriAI that gives developers one interface for calling models from OpenAI, Anthropic, Gemini, Bedrock, Azure, Ollama and dozens of other providers. Instead of learning a different SDK, auth pattern and response shape for every vendor, you write one call and swap the model string when you want to change backends.

The problem it solves is vendor lock-in. Teams that build on a single provider often discover that switching models means rewriting request builders, error handling and streaming logic across an entire codebase. LiteLLM puts a translation layer underneath your code so the business logic stays the same while the underlying model changes in a config file.

Two ways to run it cover most needs. The Python SDK drops into an existing app as a package, while the gateway (proxy server) runs as a self-hosted service that centralizes virtual keys, budgets and logging for a whole team. The trade-off is that the gateway adds infrastructure to maintain, and the open-source version expects you to host and upgrade it yourself.

Getting Started

1. Install the package with uv add litellm or pip install litellm, then set your provider API keys as environment variables.
2. For the SDK path, import completion and call it with a model string such as openai/gpt-4o or anthropic/claude-sonnet-4, passing your messages list.
3. For the gateway path, run litellm --model gpt-4o or deploy the Docker image, then point any OpenAI client at the local base URL.
4. Add a Router config when you want retries and fallbacks across multiple deployments, and connect Postgres and Redis if you run more than one gateway instance.
5. Track usage through the admin dashboard or a callback, and set budgets per key, user or team as traffic grows.