majiayu000/litellm-rs

# litellm-rs 一个用 Rust 编写的高性能 AI Gateway - 使用 OpenAI 格式调用 100+ LLM API。 [](https://crates.io/crates/litellm-rs) [](https://docs.rs/litellm-rs) [](https://opensource.org/licenses/MIT) ## 功能特性 - **100+ AI Providers** - OpenAI, Anthropic, Google, Azure, AWS Bedrock 等 - **OpenAI 兼容 API** - OpenAI SDK 的直接替代品 - **高性能** - 10,000+ 请求/秒，<10ms 路由开销 - **智能路由** - 负载均衡、故障转移、成本优化 - **企业级就绪** - 认证、限流、缓存、可观测性 ## 快速开始 ### 作为库 (API 集成) ``` use litellm_rs::{completion, user_message, system_message}; #[tokio::main] async fn main() -> Result<(), Box> { // Set your API key std::env::set_var("OPENAI_API_KEY", "sk-..."); let response = completion( "gpt-4", vec![ system_message("You are a helpful assistant."), user_message("Hello!"), ], None, ).await?; println!("{}", response.choices[0].message.content.as_ref().unwrap()); Ok(()) } ``` ### 作为 Gateway 服务器 ``` # 安装 cargo install litellm-rs # 运行 gateway --config config/gateway.yaml ``` ## 安装 ``` # Full gateway with SQLite + Redis (default) [dependencies] litellm-rs = "0.3" # API-only - lightweight, no actix-web/argon2/aes-gcm/clap [dependencies] litellm-rs = { version = "0.3", default-features = false } # API-only with metrics [dependencies] litellm-rs = { version = "0.3", default-features = false, features = ["lite"] } # Gateway server without storage [dependencies] litellm-rs = { version = "0.3", default-features = false, features = ["gateway"] } ``` ## 支持的提供商 | Provider | Chat | Embeddings | Images | Audio | |----------|------|------------|--------|-------| | OpenAI | ✅ | ✅ | ✅ | ✅ | | Anthropic | ✅ | - | - | - | | Google (Gemini) | ✅ | ✅ | ✅ | - | | Azure OpenAI | ✅ | ✅ | ✅ | ✅ | | AWS Bedrock | ✅ | ✅ | - | - | | Google Vertex AI | ✅ | ✅ | ✅ | - | | Groq | ✅ | - | - | ✅ | | DeepSeek | ✅ | - | - | - | | Mistral | ✅ | ✅ | - | - | | Cohere | ✅ | ✅ | - | - | | OpenRouter | ✅ | - | - | - | | Together AI | ✅ | ✅ | - | - | | Fireworks AI | ✅ | ✅ | - | - | | Perplexity | ✅ | - | - | - | | Replicate | ✅ | - | ✅ | - | | Hugging Face | ✅ | ✅ | - | - | | Ollama | ✅ | ✅ | - | - | | 以及 80+ 更多... | | | | | ## 环境变量 ``` # Provider API Keys OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-ant-... GOOGLE_API_KEY=... AZURE_OPENAI_API_KEY=... AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... GROQ_API_KEY=... DEEPSEEK_API_KEY=... # 可选 LITELLM_VERBOSE=true # Enable verbose logging ``` ## 示例 ### 多 Provider 路由 ``` use litellm_rs::{completion, user_message}; // Automatically routes to the right provider based on model name let openai = completion("gpt-4", vec![user_message("Hi")], None).await?; let anthropic = completion("anthropic/claude-3-opus", vec![user_message("Hi")], None).await?; let google = completion("gemini/gemini-pro", vec![user_message("Hi")], None).await?; let bedrock = completion( "bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0", vec![user_message("Hi")], None, ) .await?; ``` ### Embeddings ``` use litellm_rs::{embedding, embed_text}; // Single text let embedding = embed_text("text-embedding-3-small", "Hello world").await?; // Batch let embeddings = embedding( "text-embedding-3-small", vec!["Hello", "World"], None, ).await?; ``` ### 流式传输 ``` use litellm_rs::{completion_stream, user_message}; use futures::StreamExt; let mut stream = completion_stream( "gpt-4", vec![user_message("Tell me a story")], None, ).await?; while let Some(chunk) = stream.next().await { if let Ok(chunk) = chunk { print!("{}", chunk.choices[0].delta.content.unwrap_or_default()); } } ``` ## 性能 - **吞吐量**: 10,000+ 请求/秒 - **延迟**: <10ms 路由开销 - **内存**: ~50MB 基础占用 - **并发**: 基于 Tokio 的完全异步 ## 文档 - [API 文档](https://docs.rs/litellm-rs) - [配置指南](./config/gateway.yaml.example) - [示例](./examples/) ## 安全 请参阅 [SECURITY.md](./SECURITY.md) 了解安全策略和漏洞报告。 ## 许可证 MIT 许可证 - 详情请参阅 [LICENSE](./LICENSE)。 ## 致谢 灵感来源于 [LiteLLM](https://github.com/BerriAI/litellm) (Python)。

标签：Actix-web, AI网关, Anthropic, API管理, API集成, AWS Bedrock, Azure OpenAI, CIS基准, GPT-4, LLM API, OpenAI兼容, Redis, Rust, Rust库, SQLite, 反向代理, 可观测性, 可视化界面, 大模型开发工具, 大语言模型蜜罐, 失败转移, 搜索引擎查询, 统一接口, 缓存, 网络流量审计, 负载均衡, 通知系统, 通知系统, 配置错误