A little while ago I stumbled onto something from Cloudflare that genuinely made my jaw drop. They released an endpoint called /crawl inside their Browser Rendering service that literally takes a URL, crawls the entire site page by page on its own, and returns the content in whatever format you want. One single API call.
When I first saw it I thought “this is too good to be true”. So I started digging - read the docs, tested it for real - and I’m going to tell you what I actually found, the good and the not so good.
What is Browser Rendering and where does /crawl fit in
Before getting into the endpoint itself, you need to understand what Cloudflare’s Browser Rendering actually is, because /crawl lives inside that service.
Browser Rendering is basically a browser that Cloudflare runs on their own infrastructure. You send it requests, it executes JavaScript, loads the page the way a real browser would, and returns the result. Up until now it was mainly used with Puppeteer or Playwright directly from a Cloudflare Worker, which meant you had to write code.
The /crawl endpoint is the new thing, currently in open beta, and it’s the no-code version of all of that. You tell it what you want and it does it. No scripts, no spiders, no setup.
It’s designed especially for cases like these:
- RAG (Retrieval-Augmented Generation): when you want to feed a language model with information from a specific website.
- Documentation ingestion: for example, grabbing an entire library’s docs and processing them.
- Content snapshots: taking a point-in-time capture of what’s on a site.
What it is not - and this is important to be clear about from the start - is a scraper for real-time monitoring. If you need to check a product price every 30 seconds, this isn’t for you.
How it works under the hood
The process is asynchronous, and you need to keep that in mind because otherwise it feels like something is broken.
When you make the call, Cloudflare creates a job on their servers, starts crawling the site you pointed it at, and when it’s done (which can take a while, fair warning), you can check the results using another endpoint with the job ID.
The result comes back with each page’s content in the format you requested: HTML, Markdown, or structured JSON. Markdown is especially handy if you’re going to pass it to an LLM afterwards, since it’s a clean, easy-to-process format.
Step 1: Get your Account ID and create the API Key
The first things you need are your Cloudflare Account ID and an API Key with the right permissions.
Account ID: Head to the Cloudflare dashboard at dash.cloudflare.com, go to Workers & Pages, and you’ll see it right there in the right sidebar. Copy it and keep it somewhere because you’ll need it in a moment.
API Key: Go to dash.cloudflare.com/profile/api-tokens and create a new token. The permission you need is this one:
Account > Browser Rendering > EditWithout that permission you won’t be able to make any calls to the endpoint. Once it’s created, copy the token too - they’ll only show it to you once.
Step 2: Start the crawl
With the Account ID and token in hand, it’s time to make the call. The endpoint is:
POST https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/browser-rendering/crawlA basic example with curl looks like this:
curl -X POST "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/browser-rendering/crawl" \
-H "Authorization: Bearer YOUR_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"limit": 10,
"depth": 1
}'That’s the bare minimum. The response you get back will look something like this:
{
"success": true,
"result": {
"id": "abc123def456",
"status": "running"
}
}Save that id because you’ll need it to retrieve the results.
Step 3: Configure the call to your needs
This is where it gets interesting, because there are a lot of parameters you can tweak. What I did was hand the documentation to ChatGPT along with the site I wanted to scrape and asked it to generate the full request body. No magic to it and it saves a ton of time.
These are the most useful parameters you can include in the body:
url (required): The URL where the crawl starts.
limit: The maximum number of pages to process. On the free plan you can’t go above 100 per call.
depth: How many levels deep you want it to go. With 1 it only visits pages directly linked from the starting URL. With 2 it also follows the links on those pages, and so on.
render: Defaults to true, meaning it executes JavaScript like a real browser. If the site you’re scraping is static (like a blog or docs site with no JS), you can set this to false and it’ll run significantly faster.
includePatterns / excludePatterns: Glob pattern arrays to control which pages to include or skip. For example, if you only want blog pages:
"includePatterns": ["/blog/*"]Or if you want to skip the search section and maps:
"excludePatterns": ["/search*", "/maps*"]rejectResourceTypes: Tells it not to load certain resource types while browsing. If you don’t care about images or fonts, block them here and the whole process runs leaner:
"rejectResourceTypes": ["image", "font", "stylesheet"]responseFormat: The output format. You can ask for json, markdown, html, or multiple at once.
A more complete example could look like this:
{
"url": "https://docs.example.com",
"limit": 100,
"depth": 2,
"render": false,
"excludePatterns": ["/search*", "/changelog*"],
"rejectResourceTypes": ["image", "font"],
"responseFormat": ["markdown", "json"]
}Step 4: AI-powered structured extraction (the most powerful part)
This is what caught my attention most when I first discovered it. You can pass an AI prompt and a JSON schema so that Cloudflare, using Workers AI under the hood, extracts the data already structured for you.
Imagine you want to pull product data from an online store. The request body would look something like this:
{
"url": "https://example-shop.com/products",
"limit": 50,
"responseFormat": ["json"],
"aiOptions": {
"prompt": "Extract the information from each product you find on the page",
"jsonSchema": {
"type": "object",
"properties": {
"products": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": { "type": "string" },
"price": { "type": "number" },
"description": { "type": "string" },
"in_stock": { "type": "boolean" }
}
}
}
}
}
}
}The result comes back with your data already in the format you asked for. It’s not perfect - more on that in the conclusion - but for a first pass without writing a single CSS selector, it’s pretty solid.
Step 5: Check the status and collect the results
Since the process is asynchronous, you need to poll the status endpoint until it’s done:
curl -X GET "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/browser-rendering/crawl/YOUR_JOB_ID" \
-H "Authorization: Bearer YOUR_API_TOKEN"The response will tell you where things stand:
{
"result": {
"id": "abc123def456",
"status": "running",
"total": 100,
"finished": 47
}
}Once status changes to finished, the full results are ready. They look like this:
{
"result": {
"status": "finished",
"total": 100,
"finished": 100,
"records": [
{
"url": "https://example.com/page-1",
"metadata": {
"title": "Page title",
"status": 200
},
"markdown": "# Title\n\nPage content...",
"html": "<html>...</html>"
}
]
}
}Results stay available for 14 days after the job finishes, so no rush grabbing them - just don’t forget about them either.
How long it actually takes and what I found
Here’s the part that’s going to surprise you, and not in a good way.
In my test I took a site with a list of video game items, capped it at 100 pages, and it took almost a full hour to finish. An hour for 100 pages. I wasn’t sitting there waiting for it, it runs on its own, but if you compare that to building a scraper with Scrapy or even BeautifulSoup, the difference is pretty stark.
The results weren’t bad though, all things considered. For a beta that used AI to detect the fields, it picked up a decent amount of what I asked for. Not with the precision you’d get writing selectors by hand, but for a zero-code first pass it was respectable.
Pricing and limits
On the free plan (Workers Free) you get:
- 5 crawl jobs per day maximum
- Each job can process up to 100 pages, no more
If you need more, the paid plan (Workers Paid, $5/month) gives you 10 hours of browser time per month. Once you go over those 10 hours, the rate is $0.09 per additional hour. Not expensive if you’re using it occasionally.
Jobs have a maximum execution time of 7 days, which is more than enough for any normal site.
The CAPTCHA problem and anti-bot protections
This is something worth keeping in mind: the /crawl endpoint cannot bypass anti-bot protections. And here’s the irony: Cloudflare themselves sell those protections to their customers. So if a site has Cloudflare’s Bot Management enabled or has CAPTCHAs in place, the crawler simply won’t be able to process it.
Which makes sense logically, but it means there are sites you just won’t be able to scrape this way. If you land on a page and see Cloudflare’s “Verifying you are human” screen, forget it.
When it makes sense and when it doesn’t
After testing it, here’s what I’ve landed on.
It makes sense when you’re building a RAG pipeline, you need to ingest a website’s documentation to feed an LLM, or you want to capture a site’s content without dealing with code. The Workers AI integration for structured extraction is also interesting for cases where the site isn’t too complex.
It doesn’t make sense when you need highly precise structured data, when the site has anti-bot protection, or when response time matters. For those cases, doing it manually with Python, Playwright, or a Scrapy spider is still the better option. You’ll finish faster and have more control.
Personally, I’m going back to the manual approach. Not because this is bad, but because what I typically need requires more precision and less waiting time. But if you’ve never done scraping and have a simple use case like RAG, it’s absolutely worth checking out.
Conclusion
Cloudflare’s /crawl endpoint is a genuinely useful tool for the right use cases. The fact that a network infrastructure company shipped a crawler with AI built in and exposed it as a REST API with zero code required is impressive. But it’s still in beta, the times are slow, and it’s not the silver bullet that makes knowing how to scrape irrelevant.
If you try it, let me know in the comments how it went. And if anything in the steps wasn’t clear, just ask.
What do you think?
Leave your opinion, question or suggestion. Comments are synced with GitHub Discussions .