llm-serving
21 progetti condividono questo topic GitHub
Le linee collegano membri che sono misurabilmente correlati tra loro. La dimensione dei punti riflette le stelle.
🧬 Membri
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
★ 95.7k
ray
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for…
★ 44.1k
llm-action
★ 25.2k
TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and…
★ 15k
OpenLLM
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
★ 12.6k
BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model…
★ 8.9k
ai-infra-book
《深入理解 AI Infra：量化分析与系统设计》（李博杰…
★ 7.1k · GitHub ↗
gpustack
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU…
★ 5.9k
lorax
Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
★ 3.8k
FastDeploy
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
★ 3.7k
vllm-ascend
Community maintained hardware plugin for vLLM on Huawei Ascend
★ 3k
chitu
High-performance inference framework for large language models, focusing on efficiency, flexibility, and…
★ 3k
MoBA
MoBA: Mixture of Block Attention for Long-Context LLMs
★ 2.2k
aici
AICI: Prompts as (Wasm) Programs
★ 2.1k
kvcached
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
★ 2k · GitHub ↗
parallax
Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere
★ 1.4k
halogen-flash-server
Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models,…
★ 1.1k · GitHub ↗
helix
♾️ Private Agent Fleet with Spec Coding. Each agent gets their own GPU-accelerated desktop. Run Claude,…
★ 793
LLM-FineTuning-Large-Language-Models
LLM (Large Language Model) FineTuning
★ 578
sndr_core_engine
SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere…
★ 133
fak
Agentic Runtime
★ 42 · GitHub ↗
🔗 Famiglie affini
Misurato dai temi di GitHub condivisi da entrambi i progetti, ponderato in base a quanto è raro ciascun tema.