llm-inference

80 projets partagent ce topic GitHub

Les traits relient les membres réellement apparentés entre eux. La taille des points suit les étoiles.

🧬 Membres

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for…

Find secrets with Gitleaks 🔑

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

High-speed Large Language Model Serving for Local Deployment

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS,…

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model…

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent…

turbo-fieldfare

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

《深入理解 AI Infra：量化分析与系统设计》（李博杰…

★ 7.1k · GitHub ↗

FlashInfer: Kernel Library for LLM Serving

Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and…

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU…

Awesome-LLM-Inference

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4,…

Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai

Optimizing inference proxy for LLMs

GenerativeAIExamples

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server and Mac app…

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

AI-Engineer-Headquarters

A collection of scientific methods, processes, algorithms, and systems to build stories & models.

ai-engineering-interview-questions

Your Cheat Sheet for AI Engineering Interview – Questions and Answers.

distributed-llama

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More…

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).

An open coding agent for your terminal, built by a community collective rather than a company. Bring your own…

The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect…

AICI: Prompts as (Wasm) Programs

Project Maya - run GLM-5.3-Flash (321B MoE), DeepSeek-V4.1-Flash-1017 (748B MoE) and MiniMax-M3.1-Flash (428B…

★ 2.1k · GitHub ↗

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

★ 2k · GitHub ↗

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper`…

Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your…

Ultrafast serverless GPU inference, sandboxes, and background jobs

react-native-executorch

Declarative way to run AI models in React Native on device, powered by ExecuTorch.

Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch,…

★ 1.6k · GitHub ↗

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators.…

LLM Inference Engine for Metal, CUDA and Vulkan.

★ 1.5k · GitHub ↗

Run local LLMs like llama, deepseek-distill, kokoro and more inside your browser

awesome-ai-web-search

List of software that allows searching the web with the assistance of AI:…

awesome-hacking-lists

A curated collection of top-tier penetration testing tools and productivity utilities across multiple…

LLMs as Copilots for Theorem Proving in Lean

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

time-to-first-token

A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization,…

★ 1k · GitHub ↗

Open-source VMs-as-a-service

LLM-PowerHouse-A-Curated-Guide-for-Large-Language-Models-with-Custom-Training-and-Inferencing

LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for…

Minimalist web-searching platform with an AI assistant that runs directly from your browser. Demo:…

LLM-FineTuning-Large-Language-Models

LLM (Large Language Model) FineTuning

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM…

Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default,…

[NeurIPS 2026 Spotlight] Fast, lossless LLM inference via dual-view diffusion decoding.

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs…

Explore the unknown, build the future, own your data.

LLMs and Machine Learning done easily

Context-Engine MCP - Agentic Context Compression Suite

Efficient LLM Inference over Long Sequences

embedding_studio

Embedding Studio is a framework which allows you transform your Vector Database into a feature-rich Search…

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models,…

PyTorch library for cost-effective, fast and easy serving of MoE models.

Efficient and general syntactical decoding for Large Language Models

Monocle is a framework for tracing GenAI app code. This repo contains implementation of Monocle for GenAI…

Drop-in prompt compression for production LLM apps. Cut your token bill 40-60% without changing your code.…

🌱 EcoLogits tracks the energy consumption and environmental footprint of using generative AI models…

On-device LLM Inference Powered by X-Bit Quantization

Run generative AI models in sophgo BM1684X/BM1688

llm-inference-capacity-handbook

大模型推理案头手册（开源版）：给定 GPU 算力，一个模型能扛多少 QPS？三层模型…

★ 185 · GitHub ↗

deepseek-local-guide-ru

🐋 Полный гайд на русском: как установить и запустить DeepSeek…

★ 157 · GitHub ↗

sndr_core_engine

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere…

free-llm-api-hub

A continuously-verified dataset of free-tier & trial-credit LLM APIs for developers. Every entry dated,…

★ 109 · GitHub ↗

High-performance Lemonade alternative for AMD Strix Halo & Radeon — latest MoE models (Ling-3.0-Flash, Qwen…

★ 94 · GitHub ↗

AI-ML Companion: an interactive platform to learn AI & ML by watching it work - 28 tracks, 400+ modules, live…

llama.cpp optimized for NVIDIA Tesla V100 (SM70): Qwen3.8-Flash-Next with MTP (prefill 3402 tok/s at 200K…

★ 59 · GitHub ↗

Open-source infrastructure for the full inference lifecycle: deploy, observe, scale, optimize, and safely…

Zero-code LLM security & observability proxy. Real-time prompt injection detection, PII scanning, and cost…

Bounded, inspectable LLM inference pipelines from declared YAML — runs offline against Ollama or any…

Agentic Runtime

★ 42 · GitHub ↗

Native AI inference for PHP 8.3+ - run ONNX, GGUF (llama.cpp) and RubixML models directly in your PHP process…

🔗 Familles voisines

llm-serving 21 local-llm 60 vllm 48 mlops 67 inference 81 cuda 82 llama2 40 mixture-of-experts 16

Mesuré à partir des topics GitHub communs aux deux projets, pondérés par leur rareté.