aws-samples/sagemaker-genai-hosting-examples

# Amazon SageMaker 上的生成式 AI 推理示例 此代码仓库包含了一系列利用 SageMaker Inference 进行热门大型语言模型 (LLM) 优化部署的示例。由于模型体积庞大、硬件利用率低下，以及在具有多个并发用户的生产级环境中扩展 LLM，托管 LLM 面临着诸多挑战。 SageMaker Inference 是一种高性能且通用的托管选项，提供了多种配置供您高效地托管 LLM。在此代码仓库中，我们展示了如何使用不同的 SageMaker Inference 选项，例如 Real-Time Inference（适用于低延迟、高吞吐量的使用场景）和 Asynchronous Inference（适用于近实时/批处理使用场景），并将它们与 [DJL Serving](https://github.com/deepjavalibrary/djl-serving) 和 [Text Generation Inference](https://github.com/huggingface/text-generation-inference) 等模型服务器集成。我们展示了如何通过优化这些不同的模型服务栈以及探索硬件选项（例如 [Inferentia2](https://aws.amazon.com/blogs/machine-learning/achieve-high-performance-with-lowest-cost-for-generative-ai-inference-using-aws-inferentia2-and-aws-trainium-on-amazon-sagemaker/) 与 Amazon SageMaker 的集成）来进行性能调优。 ## 内容 - [Mistral](./Mistral/) - [Mixtral](./Mistral/) - [Falcon](./Falcon/) - [Flan](./FlanT5/) - [Llama2](./Llama2/) - [Llama3](./Llama3/) - [Open-Llama](./Open-Llama/) - [Zephyr](./Zephyr/) - [CodeGen](./Codegen25/) - [CodeLlama](./CodeLlama/) - [Swiss AI Initiative Apertus](./01-models/Swiss-AI/Apertus/) ## 附加资源 - [大型模型推理容器简介](https://aws.amazon.com/blogs/machine-learning/boost-inference-performance-for-llms-with-new-amazon-sagemaker-containers/) - [LLM 推理优化工具包](https://aws.amazon.com/blogs/machine-learning/achieve-up-to-2x-higher-throughput-while-reducing-costs-by-50-for-generative-ai-inference-on-amazon-sagemaker-with-the-new-inference-optimization-toolkit-part-1/) - [大型模型推理容器调优指南](https://docs.djl.ai/docs/serving/serving/docs/lmi/tuning_guides/deepspeed_tuning_guide.html) - [使用 Amazon SageMaker 进行 Text Generation Inference](https://aws.amazon.com/blogs/machine-learning/announcing-the-launch-of-new-hugging-face-llm-inference-containers-on-amazon-sagemaker/) - [使用 LMI 进行服务端批处理优化](https://aws.amazon.com/blogs/machine-learning/improve-throughput-performance-of-llama-2-models-using-amazon-sagemaker/) - [通用 SageMaker 托管示例代码仓库](https://github.com/aws-samples/sagemaker-hosting) - [SageMaker 托管博客系列](https://aws.amazon.com/blogs/machine-learning/model-hosting-patterns-in-amazon-sagemaker-part-1-common-design-patterns-for-building-ml-applications-on-amazon-sagemaker/) - [轻松部署和管理数百个 LoRA 适配器](https://aws.amazon.com/blogs/machine-learning/easily-deploy-and-manage-hundreds-of-lora-adapters-with-sagemaker-efficient-multi-adapter-inference/) ## 安全 ## 许可证 本库采用 MIT-0 许可证授权。请参阅 [LICENSE](./LICENSE) 文件。

标签：请求拦截, 逆向工具