web-crawling
6 projects share this GitHub topic
Lines connect members that are measurably related to each other. Dot size reflects stars.
🧬 Members
crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run…
★ 86.4k
crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript…
★ 26.3k
moli
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
★ 21.2k
crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data…
★ 9.6k
brightdata-mcp
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
★ 2.7k
Craw4LLM
Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"
★ 666
🔗 Related families
Measured from GitHub topics shared by both projects, weighted by how rare each topic is.