lmdeploy
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
LMDeploy is a toolkit for compressing, quantizing and serving LLMs with high request throughput via its TurboMind engine.
From the README
- \[2024/09\] LMDeploy PyTorchEngine achieves 1.3x faster on Llama3-8B inference by introducing CUDA graph
- \[2024/08\] LMDeploy is integrated into modelscope/swift as the default accelerator for VLMs inference
- \[2024/07\] Support Llama3.1 8B, 70B and its TOOLS CALLING
- \[2024/07\] Support InternVL2 full-series models, InternLM-XComposer2.5 and function call of InternLM2.5
- \[2024/06\] PyTorch engine support DeepSeek-V2 and several VLMs, such as CogVLM2, Mini-InternVL, LlaVA-Next
- \[2024/05\] Balance vision model when deploying VLMs with multiple GPUs
Excerpts from the project README on GitHub. Copyright and licensing remain with the respective authors.
Claim its page: indexed whatever its rank, translated into six languages, and enriched with what you write yourself.
Get an email alert on its next release or when it starts trending — never miss the moment.
Free · no card · unsubscribe anytimeLMDeploy is a toolkit for compressing, deploying, and serving LLMs.
lmdeploy has 8k stars on GitHub. It has been forked 734 times. lmdeploy is written mainly in Python. It has been in active development since 2023. lmdeploy is available under the Apache-2.0 license. Its main topics are codellama, cuda-kernels, deepspeed, fastertransformer.
Read the full guideWhat is lmdeploy?
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
Is lmdeploy open source?
lmdeploy is an open-source project. It is released under the Apache-2.0 license.
Is lmdeploy free?
Yes. lmdeploy is free and open source — you can use, modify and self-host it.
What license does lmdeploy use?
lmdeploy is available under the Apache-2.0 license.
What language is lmdeploy written in?
lmdeploy is written mainly in Python.
Add this live badge to your README — your GitHub stars and directory rank, refreshed daily.
[](https://olud.ai/project/internlm-lmdeploy.html)
Measured from GitHub topics shared by both projects, weighted by how rare each topic is.