llm-inference

76 progetti condividono questo topic GitHub

Le linee collegano membri che sono misurabilmente correlati tra loro. La dimensione dei punti riflette le stelle.

🧬 Membri

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for…

Find secrets with Gitleaks 🔑

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

High-speed Large Language Model Serving for Local Deployment

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model…

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent…

FlashInfer: Kernel Library for LLM Serving

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework…

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No…

Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.

Awesome-LLM-Inference

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4,…

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU…

Superduper: End-to-end framework for building custom AI applications and agents.

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and…

Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai

RuVector is a High Performance, Real-Time, Self-Learning Ai, Vector GNN, Memory DB built in Rust.

Optimizing inference proxy for LLMs

GenerativeAIExamples

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query,…

distributed-llama

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More…

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).

An open coding agent for your terminal, built by a community collective rather than a company. Bring your own…

AICI: Prompts as (Wasm) Programs

The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect…

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper`…

Ultrafast serverless GPU inference, sandboxes, and background jobs

react-native-executorch

Declarative way to run AI models in React Native on device, powered by ExecuTorch.

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators.…

Run local LLMs like llama, deepseek-distill, kokoro and more inside your browser

awesome-ai-web-search

List of software that allows searching the web with the assistance of AI:…

awesome-hacking-lists

A curated collection of top-tier penetration testing tools and productivity utilities across multiple…

LLMs as Copilots for Theorem Proving in Lean

Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your…

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

A highly optimized LLM inference acceleration engine for Llama and its variants.

Llama 3+ inference in pure Java

Open-source VMs-as-a-service

LLM-PowerHouse-A-Curated-Guide-for-Large-Language-Models-with-Custom-Training-and-Inferencing

LLM-PowerHouse: Unleash LLMs' potential through curated tutorials, best practices, and ready-to-use code for…

LLMFlows - Simple, Explicit and Transparent LLM Apps

Pure Rust Inference Engine

Minimalist web-searching platform with an AI assistant that runs directly from your browser. Demo:…

LLM-FineTuning-Large-Language-Models

LLM (Large Language Model) FineTuning

Local AI Assistant on your phone

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM…

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market…

✨ AI interface for tinkerers (Ollama, Haystack RAG, Python)

Efficient AI Inference & Serving

Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default,…

Explore the unknown, build the future, own your data.

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs…

LLMs and Machine Learning done easily

Context-Engine MCP - Agentic Context Compression Suite

Efficient LLM Inference over Long Sequences

embedding_studio

Embedding Studio is a framework which allows you transform your Vector Database into a feature-rich Search…

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models,…

AI-powered cybersecurity chatbot designed to provide helpful and accurate answers to your…

Efficient and general syntactical decoding for Large Language Models

A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with…

PyTorch library for cost-effective, fast and easy serving of MoE models.

Monocle is a framework for tracing GenAI app code. This repo contains implementation of Monocle for GenAI…

Drop-in prompt compression for production LLM apps. Cut your token bill 40-60% without changing your code.…

CPU inference for the DeepSeek family of large language models in C++

On-device LLM Inference Powered by X-Bit Quantization

🌱 EcoLogits tracks the energy consumption and environmental footprint of using generative AI models…

PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving,…

sndr_core_engine

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere…

Native AI inference for PHP 8.3+ - run ONNX, GGUF (llama.cpp) and RubixML models directly in your PHP process…

★ 56 · GitHub ↗

Zero-code LLM security & observability proxy. Real-time prompt injection detection, PII scanning, and cost…

★ 52 · GitHub ↗

Bounded, inspectable LLM inference pipelines from declared YAML — runs offline against Ollama or any…

★ 45 · GitHub ↗

🔗 Famiglie affini

llm-serving 23 llama2 56 inference 94 cuda 90 mlops 103 model-serving 26 vllm 47 llm-training 20

Misurato dai temi di GitHub condivisi da entrambi i progetti, ponderato in base a quanto è raro ciascun tema.