Firecrawl vs Crawl4AI: Hosted API or Your Own Crawler
November 18, 2026

Firecrawl vs Crawl4AI comes down to one question: do you want to call a web data API, or run a crawler? Both turn web pages into clean Markdown and structured JSON for LLMs. They start from opposite ends.
- Firecrawl is an API product first. You get a key, call an endpoint, and Firecrawl runs the browsers and proxies. The core is open source under AGPL-3.0, but the self-hosted version leaves out several of the newer features.
- Crawl4AI is a Python library first. You install it, it drives a real browser on your machine, and you control every setting. It's Apache-2.0 with an attribution requirement. A hosted Crawl4AI Cloud now exists too.
So the short answer:
- No Python, no servers, want it working in ten minutes? Firecrawl.
- Python team, high volume, data that shouldn't leave your network? Crawl4AI.
- Building a closed-source product on a modified crawler? Read the licence section before either.
Everything below comes from each project's repository, docs and pricing page as they read in November 2026. There are no speed or success-rate numbers here. Each vendor publishes its own, and none of them is neutral.
Quick Comparison
| Firecrawl | Crawl4AI | |
|---|---|---|
| Starting point | Hosted API | Python library |
| Written in | TypeScript | Python |
| Core licence | AGPL-3.0 (SDKs and some UI are MIT) | Apache-2.0 plus an attribution requirement |
| Hosted option | Firecrawl Cloud | Crawl4AI Cloud |
| Self-hosting | Docker Compose stack; some features are Cloud-only | Library in your process, or a Docker server |
| Output | Markdown, HTML, screenshots, JSON | Markdown, filtered "fit" Markdown, JSON, screenshots, PDFs |
| Structured extraction | JSON format on scrape, LLM-backed | CSS, XPath and regex schemas without an LLM, or LLM extraction |
| Web search | Search endpoint | Cloud only |
| MCP | Official MCP server | MCP on the Docker server and on Cloud |
| SDKs | Python, Node.js, Go, Rust, Java, .NET, PHP, Ruby, Elixir | Python library; HTTP API for everything else |
| Cost model | Credits per page, monthly plans | No licence fee for the library; credits on Cloud |
| GitHub stars | About 203,000 | About 87,000 |
Star counts are read from each GitHub repository as of November 2026.
What Each One Is
Firecrawl: a web data API
Firecrawl describes itself as a way to "search, scrape, and interact with the web at scale." Its README lists three core endpoints:
- Search returns web results with the full page content attached.
- Scrape converts one URL to Markdown, HTML, a screenshot or structured JSON.
- Interact scrapes a page, then lets you drive it with prompts or code.
Around those sit Crawl (every URL on a site in one request), Map (list a site's URLs), Batch Scrape, Parse (PDFs and DOCX into Markdown or JSON), Monitor (alerts when a page changes) and Agent (describe the data you need and let it gather it).
There's also Alexandria, a library of third-party data providers you reach through the same key. The README is explicit that it runs on Firecrawl Cloud and isn't part of the self-hosted server.
You call all of this over HTTP, from a CLI, or through SDKs for nine languages.
Crawl4AI: a crawler you run
Crawl4AI is a Python package. You pip install it, run a one-time setup that installs a browser, and call AsyncWebCrawler from your own code. The page loads in a real browser on your machine. Chromium, Firefox and WebKit are supported.
What you get back is a result object with Markdown, links, media and metadata. The library's features are mostly knobs on that process:
- Markdown generation, including "fit Markdown" that filters out menus, footers and boilerplate.
- Deep crawling with breadth-first, depth-first and best-first strategies, plus resume after a crash.
- Adaptive crawling, which stops once it has gathered enough to answer a query.
- Browser control: persistent profiles with saved logins, sessions, proxies with rotation, hooks, custom JavaScript, anti-detection options and an undetected-browser adapter.
If you don't want it inside your Python process, there's a Docker server with a REST API, a dashboard and a playground.
There's now a third way to use it. The README now opens with Crawl4AI Cloud, a hosted API from the same team with /scrape, /extract and /search endpoints, plus a SERP endpoint marked beta and an answer endpoint marked experimental.
Licence: AGPL-3.0 vs Apache-2.0 With Attribution
This is the difference people skip and regret later.
Firecrawl's core is AGPL-3.0. The README says the project is "primarily licensed" under AGPL-3.0, with the SDKs and some UI components under MIT. In practice:
Calling the hosted API, or using the MIT SDKs, puts no licence obligation on your code. Running the server unmodified inside your company is generally fine. Modifying the server and offering it to users over a network is where AGPL bites: its network clause asks you to offer those users your modified source.
Crawl4AI's LICENSE file is Apache-2.0 with an added "Attribution Requirement" section. That section says distributions, publications and public uses of the software, or derivative works, must include a credit line to the project. The README offers two ways to do it, a badge or a text credit, and calls the badge route "recommended."
Apache-2.0 is permissive: you can embed Crawl4AI in closed-source software. The attribution add-on is a small cost for most teams, but it's a real condition and it isn't part of standard Apache-2.0. If you ship software to customers, show the file to whoever approves your open-source use.
| You want to... | Firecrawl | Crawl4AI |
|---|---|---|
| Call a hosted API from a closed product | Fine | Fine |
| Embed the crawler in a closed product | Not with the AGPL core | Yes, with attribution |
| Modify it and run it as a public service | Share your changes | Yes, with attribution |
This is a reading of the licence texts, not legal advice.
Self-Hosting: What You Actually Get
Both can run on your hardware. What you end up with differs a lot.
Firecrawl self-hosted
Firecrawl's docs have a page that compares open source with Firecrawl Cloud, and it's candid:
| Capability | Self-hosted |
|---|---|
| Scrape, crawl, map and search APIs | Included |
| Fetch and Playwright rendering | Included in the default stack |
| LLM-backed extraction and formats | You connect an OpenAI-compatible provider or Ollama |
| Advanced anti-bot or specialised extraction | You run and configure a separate service |
| Agent, Browser, Interact, dashboard, enterprise controls | Not included in the default stack |
The self-hosting guide adds operational detail. The Docker Compose stack runs the API, workers, Playwright, Redis, RabbitMQ and PostgreSQL. The quickstart disables API authentication and starts without durable storage or TLS, and the guide says outright that it "is not a production architecture." You own upgrades, security, storage and recovery.
Firecrawl's own recommendation is to start with Cloud unless source access or infrastructure control is worth the work. Take that at face value: self-hosted Firecrawl is the core scraper, not the whole product.
Crawl4AI self-hosted
With Crawl4AI, self-hosted is the original product. The library has no missing tier. The self-hosting guide for the Docker server covers a REST API, a monitoring dashboard, a job queue with webhooks, and MCP endpoints.
Two things to know:
- Authentication is on by default since 0.9.0. The server needs a
CRAWL4AI_API_TOKENand binds to loopback until you set one. The guide flags this as a breaking change from 0.8.x. - Hard sites are your problem. The README's own comparison table says that on the library and your own server, JS-heavy pages and bot walls come down to "your settings, your proxies." Crawl4AI gives you the tools. It doesn't give you a proxy pool.
Web search isn't in the library either. That lives on Crawl4AI Cloud.
Extraction and Crawling
Getting structured data. Firecrawl's route is the JSON format on a scrape: you pass a schema or a prompt and a model fills it in. It's quick to set up, and the pricing page shows that JSON adds credits per page on top of the basic scrape.
Crawl4AI gives you two routes. The first uses no model at all: JsonCssExtractionStrategy, JsonXPathExtractionStrategy and RegexExtractionStrategy pull fields with selectors you define. For a site with a stable layout, like a product catalogue, that means no per-page model cost and repeatable output. The second is LLMExtractionStrategy, which works with any provider the library's model layer supports, including local models.
Crawling a whole site. Firecrawl's Crawl endpoint takes a start URL and returns every page it reaches as a job you poll, and Map gives you the URL list first. Crawl4AI's deep crawling runs in your process with a strategy you choose, filters and scorers you write, and a state you can resume.
Pages that need clicks. Firecrawl has Interact on Cloud. Crawl4AI has sessions, hooks and JavaScript execution in the library. If the job is mostly clicking through a logged-in app, a browser automation tool may suit better than either, which our Selenium web scraping guide covers.
| Need | Better fit |
|---|---|
| One URL to Markdown, right now | Firecrawl |
| Fields from a stable page layout, at volume | Crawl4AI's selector strategies |
| Fields from messy, varied pages | Either, with an LLM |
| Search results plus page content in one call | Firecrawl, or Crawl4AI Cloud |
| A crawl you tune link by link | Crawl4AI |
| PDF and DOCX parsing as an endpoint | Firecrawl |
MCP and Coding Agents
Both plug into agents.
Firecrawl publishes an official MCP server that runs with one npx command and an API key, plus a CLI and skill installer for coding agents such as Claude Code. Its docs also describe a local MCP setup that points at a self-hosted instance.
Crawl4AI's Docker server exposes MCP over SSE and WebSocket, so an agent can use a crawler running on your own machine with no third party involved. Crawl4AI Cloud has a hosted MCP server over streamable HTTP.
If your goal is "my agent should be able to read the web," all of these work. If page content shouldn't pass through a vendor, use one of the self-hosted routes: Crawl4AI's own server, or a self-hosted Firecrawl with its local MCP setup.
Cost: Credits vs Your Own Machines
Firecrawl sells credits. Per its pricing page, one credit is one page on a basic scrape or crawl, search is billed per ten results, and Interact is billed per browser minute. There's a no-cost plan with 1,000 credits a month and no card, then monthly plans that raise credits, concurrency and rate limits. A scrape that returns nothing isn't charged; a page that answers with an error status still costs a credit.
Crawl4AI the library has no licence fee. You pay for:
- Servers with enough memory to run browsers.
- Proxies, if your targets block datacentre traffic.
- Model calls, if you use LLM extraction.
- Engineering time for retries, scaling and upkeep.
Crawl4AI Cloud uses credits too, with 1,000 a month at no cost and paid plans above that. Its docs list what each call costs.
Which is cheaper depends on volume and on how hard your targets are. At a few thousand pages a month, either hosted option costs less than an engineer's afternoon. At millions of pages from cooperative sites, your own Crawl4AI workers usually win. At millions of pages from sites that fight back, the proxy bill decides, and you should price it before assuming self-hosting is the cheap route.
For a wider field, see our Firecrawl alternatives and Crawl4AI alternatives roundups.
A Note on Permission
Neither tool decides what you're allowed to collect. Check a site's terms and robots.txt, respect rate limits, and be careful with personal data. Tools that make blocked pages reachable don't change who owns the content.
Which Should You Pick?
Pick Firecrawl if:
- You want an API key and results today, in any language.
- You need search, crawl, parsing and page monitoring from one vendor.
- You'd rather pay per page than run browsers.
- AGPL doesn't matter because you're only calling the hosted API.
Pick Crawl4AI if:
- Your team writes Python and wants control over the browser.
- Pages or outputs shouldn't leave your infrastructure.
- You have stable page layouts and want extraction without model costs.
- You need a permissive licence to embed the crawler in your own product.
Use both if you have a large, predictable crawl and a small set of hard targets. Run the bulk on your own Crawl4AI workers and send the stubborn pages to a hosted API.
Where Taku Fits
Both tools assume you're building a pipeline. Plenty of people asking this question only want the outcome, not the crawler.
If you'd rather run a tool someone already built than wire up a crawler yourself, that's where Taku comes in. Taku Apps and Sites, built by the best AI minds. Find one, run it, and keep it in your Mac Dock. Taku is in Beta, and the Mac app is available now.
FAQ
Is Crawl4AI better than Firecrawl?
Neither is better across the board. Crawl4AI gives a Python team more control and a permissive licence, and costs only infrastructure when self-hosted. Firecrawl gives you more endpoints behind one key, SDKs in many languages, and nothing to operate.
Is Firecrawl open source?
The core is, under AGPL-3.0, with MIT-licensed SDKs. The self-hosted version covers scrape, crawl, map and search. Firecrawl's docs say Agent, Browser, Interact, the dashboard and enterprise controls aren't in the default self-hosted stack, and Alexandria runs on Cloud only.
Is Crawl4AI really free?
The library has no licence fee under Apache-2.0, with an attribution requirement in its LICENSE file. Running it still costs servers, proxies if you need them, and model calls if you use LLM extraction. Crawl4AI Cloud is a separate paid service with a no-cost monthly allowance.
Does Crawl4AI need an LLM?
No. Markdown output and selector-based extraction work without one. A model is only needed for LLM extraction or LLM-based content filtering, and you can point those at a local model.
Which is better for RAG?
Both produce Markdown suited to chunking. For a one-off import of a docs site, Firecrawl's crawl endpoint is the fastest path. For a recurring ingest you want to tune and keep in-house, Crawl4AI fits better. If you'd rather pick from a catalogue of ready-made scrapers, Apify alternatives covers that category.