web-crawler
15 projects share this GitHub topic
Lines connect members that are measurably related to each other. Dot size reflects stars.
🧬 Members
firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
★ 195.7k
crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run…
★ 86.4k
Scrapegraph-ai
Python scraper based on AI
★ 32.3k
crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript…
★ 26.3k
moli
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
★ 21.2k
crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data…
★ 9.6k
firecrawl-mcp-server
Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus…
★ 7.7k
wigolo
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys,…
★ 5.5k
ai-crawler-py
Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural…
★ 3.3k
webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust.…
★ 2.4k
crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for…
★ 1.1k · GitHub ↗
crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for…
★ 1.1k
Craw4LLM
Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"
★ 666
reader
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions,…
★ 558
pagesift
High-performance, memory-efficient Rust web crawler & content extraction engine for AI agents and RAG.…
★ 113 · GitHub ↗
🔗 Related families
Measured from GitHub topics shared by both projects, weighted by how rare each topic is.