web-crawler
13 projects share this GitHub topic
Lines connect members that are measurably related to each other. Dot size reflects stars.
🧬 Members
firecrawl
The API to search, scrape, and interact with the web at scale. 🔥
★ 157.9k
Scrapegraph-ai
Python scraper based on AI
★ 28.8k
crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript…
★ 25.1k
crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data…
★ 9.4k
firecrawl-mcp-server
🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other…
★ 7.1k
wigolo
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys,…
★ 3.8k
ai-crawler-py
Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural…
★ 3.1k
webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust.…
★ 2.1k
firecrawl-app-examples
🔥 This repository contains complete application examples, including websites and other projects, developed…
★ 794
Craw4LLM
Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"
★ 660
reader
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions,…
★ 552
crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for…
★ 495
kochat
Opensource Korean chatbot framework
★ 462
🔗 Related families
Measured from GitHub topics shared by both projects, weighted by how rare each topic is.