llm-evaluation

52 projetos partilham este topic do GitHub

Linhas conectam membros que estão mensuravelmente relacionados entre si. O tamanho do ponto reflete estrelas.

🧬 Membros

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open…

Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in…

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to…

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare…

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive…

The LLM Evaluation Framework

AI Observability & Evaluation

the LLM vulnerability scanner

Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!

A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra…

chinese-llm-benchmark

非线智能 NoneLinear - ReLE评测：中文AI大模型能力评测（持续更新）：目前已囊括374个…

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23…

🐢 Open-Source Evaluation & Testing library for LLM Agents

LLM-Engineers-Handbook

The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps…

Agent OS: Stop prompting. Start specifying. A Socratic interview gates the spec on an ambiguity score, then…

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

AI-Engineer-Headquarters

A collection of scientific methods, processes, algorithms, and systems to build stories & models.

Evaluation and Tracking for LLM Experiments and AI Agents

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

Comprehensive resources on Generative AI, including a detailed roadmap, projects, use cases, interview…

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications.…

A powerful tool for automated LLM fuzzing. It is designed to help developers and security researchers…

Trace-native CI/CD for AI agents — production failures become regression tests that block the PR.…

Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty…

Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable…

★ 1.2k · GitHub ↗

Open-source benchmark for browser AI agents on daily tasks.

The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and…

A curated, non-BS library of the best resources for building and evaluating AI agents — papers, blogs,…

A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's…

★ 953 · GitHub ↗

Trace-native CI/CD for AI agents — production failures become regression tests that block the PR.…

agent-skills-eval

A test runner for agentskills.io-style AI agent skills

The platform for continuously improving AI agents.

★ 673 · GitHub ↗

Awesome-LLM-in-Social-Science

Awesome papers involving LLMs in Social Science.

[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness,…

EnterpriseRAG-Bench

Dataset and benchmark for RAG on company internal documents.

AI equity research agent with resilient workflows, evidence-grounded RAG, versioned reports, and automated…

continuous-eval

Data-Driven Evaluation for LLM-Powered Applications

The testing platform for AI teams. Bring engineers, PMs, and domain experts together to generate tests,…

A long-horizon benchmark harness: give a coding agent a real game engine, professional conditions and time,…

llm-leaderboard

A comprehensive set of LLM benchmark scores and provider prices. (deprecated, read more in README)

Compile real-world Claude Code and Codex trajectories into verified, tradable post-training assets.

★ 281 · GitHub ↗

Evolutionary multi-agent runtime that breeds, evaluates, and improves autonomous agents across reproducible…

Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for…

★ 194 · GitHub ↗

Engineering deterministic, production-grade systems around non-deterministic LLMs — FSM, durable execution,…

ai-eval-platform

Agent evals and guardrails as Jev decisions: one request per trace, a fraction of a cent, fast enough for the…

★ 107 · GitHub ↗

Evaluate visual models on your own images, JSON Schema, and production constraints with LangGraph, Pareto…

ai-reliability-copilot

Turn a production incident into a structured 9-section LLM response (severity, root cause, mitigation,…

AI-ML Companion: an interactive platform to learn AI & ML by watching it work - 28 tracks, 400+ modules, live…

langfuse-workshop

End-to-end Langfuse workshop using a TypeScript Agent to teach the AI engineering loop: tracing, prompt…

★ 57 · GitHub ↗

awesome-llm-observability

50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to…

★ 43 · GitHub ↗

Phoenix-native LLM evaluation, prompt testing, model comparison, red teaming, and observability for Elixir

★ 41 · GitHub ↗

🔗 Familias relacionadas

llm-observability 32 evals 25 observability 79 agent-evaluation 13 llm-security 38 mlops 67 ai-observability 14 llama-index 17

Medido a partir dos tópicos do GitHub compartilhados por ambos os projetos, ponderado pela raridade de cada tópico.