web-crawler
12 projects share this GitHub topic
Lines connect members that are measurably related to each other. Dot size reflects stars.
🧬 Members
firecrawl
The context API to search, scrape, and interact with the web at scale. 🔥
★ 174.8k
Scrapegraph-ai
Python scraper based on AI
★ 30.2k
crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript…
★ 25.6k
crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data…
★ 9.5k
firecrawl-mcp-server
🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other…
★ 7.4k
wigolo
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys,…
★ 4.8k
ai-crawler-py
Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural…
★ 3.3k
webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust.…
★ 2.3k
moli
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
★ 1.6k
crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for…
★ 894
Craw4LLM
Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"
★ 664
reader
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions,…
★ 558
🔗 Related families
Measured from GitHub topics shared by both projects, weighted by how rare each topic is.