llm-inference

65 projects share this GitHub topic

Lines connect members that are measurably related to each other. Dot size reflects stars.

🧬 Members

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for…

Find secrets with Gitleaks 🔑

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

High-speed Large Language Model Serving for Local Deployment

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model…

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent…

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS,…

turbo-fieldfare

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

FlashInfer: Kernel Library for LLM Serving

Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU…

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and…

Awesome-LLM-Inference

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4,…

Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai

Optimizing inference proxy for LLMs

GenerativeAIExamples

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

AI-Engineer-Headquarters

A collection of scientific methods, processes, algorithms, and systems to build stories & models.

distributed-llama

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More…

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).

An open coding agent for your terminal, built by a community collective rather than a company. Bring your own…

The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect…

AICI: Prompts as (Wasm) Programs

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper`…

Ultrafast serverless GPU inference, sandboxes, and background jobs

react-native-executorch

Declarative way to run AI models in React Native on device, powered by ExecuTorch.

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators.…

Run local LLMs like llama, deepseek-distill, kokoro and more inside your browser

awesome-ai-web-search

List of software that allows searching the web with the assistance of AI:…

Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your…

awesome-hacking-lists

A curated collection of top-tier penetration testing tools and productivity utilities across multiple…

LLMs as Copilots for Theorem Proving in Lean

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

Open-source VMs-as-a-service

LLM-PowerHouse-A-Curated-Guide-for-Large-Language-Models-with-Custom-Training-and-Inferencing

LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for…

Minimalist web-searching platform with an AI assistant that runs directly from your browser. Demo:…

LLM-FineTuning-Large-Language-Models

LLM (Large Language Model) FineTuning

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM…

Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default,…

Explore the unknown, build the future, own your data.

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs…

LLMs and Machine Learning done easily

Context-Engine MCP - Agentic Context Compression Suite

Efficient LLM Inference over Long Sequences

embedding_studio

Embedding Studio is a framework which allows you transform your Vector Database into a feature-rich Search…

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models,…

PyTorch library for cost-effective, fast and easy serving of MoE models.

Efficient and general syntactical decoding for Large Language Models

Monocle is a framework for tracing GenAI app code. This repo contains implementation of Monocle for GenAI…

🌱 EcoLogits tracks the energy consumption and environmental footprint of using generative AI models…

On-device LLM Inference Powered by X-Bit Quantization

Drop-in prompt compression for production LLM apps. Cut your token bill 40-60% without changing your code.…

Run generative AI models in sophgo BM1684X/BM1688

sndr_core_engine

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere…

Zero-code LLM security & observability proxy. Real-time prompt injection detection, PII scanning, and cost…

Open-source infrastructure for the full inference lifecycle: deploy, observe, scale, optimize, and safely…

Bounded, inspectable LLM inference pipelines from declared YAML — runs offline against Ollama or any…

Run Ornith, Qwen, Nemotron & DeepSeek optimized on AMD Strix Halo — unified OpenAI-compatible server with…

★ 41 · GitHub ↗

Native AI inference for PHP 8.3+ - run ONNX, GGUF (llama.cpp) and RubixML models directly in your PHP process…

🔗 Related families

llm-serving 17 inference 74 llama2 41 mlops 59 vllm 35 local-llm 42 model-serving 16 transformer 59

Measured from GitHub topics shared by both projects, weighted by how rare each topic is.