crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright,
Build web scrapers to collect data from websites and automate browser tasks.
From the README
npm install crawlee playwright
- Single interface for HTTP and headless browser crawling
- Persistent queue for URLs to crawl (breadth & depth first)
- Pluggable storage of both tabular data and files
- Automatic scaling with available system resources
- Integrated proxy rotation and session management
- Lifecycles customizable with hooks
Excerpts from the project README on GitHub. Copyright and licensing remain with the respective authors.
Claim its page: indexed whatever its rank, translated into six languages, and enriched with what you write yourself.
Get an email alert on its next release or when it starts trending — never miss the moment.
Free · no card · unsubscribe anytimeCrawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright,
crawlee has 25.6k stars on GitHub. It has been forked 1.7k times. crawlee is written mainly in TypeScript. It has been in active development since 2016. crawlee is available under the Apache-2.0 license. Its main topics are apify, automation, crawler, crawling.
What is crawlee?
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright,
Is crawlee open source?
crawlee is an open-source project. It is released under the Apache-2.0 license.
Is crawlee free?
Yes. crawlee is free and open source — you can use, modify and self-host it.
What license does crawlee use?
crawlee is available under the Apache-2.0 license.
What language is crawlee written in?
crawlee is written mainly in TypeScript.
Add this live badge to your README — your GitHub stars and directory rank, refreshed daily.
[](https://olud.ai/project/apify-crawlee.html)
Measured from GitHub topics shared by both projects, weighted by how rare each topic is.