Most of a web scraping project's cost and risk is decided before a single line of code is written — by the target site, not the data you want off it. A scrape can be a two-hour job or a permanent engineering commitment, and the difference is almost never obvious from looking at the page. So before you build one or pay someone to, it's worth spending twenty minutes deciding whether the project is a green light, a proceed-with-care yellow, or a red flag you should walk away from.
This is the checklist we run on every scraping enquiry before we quote it. It won't make you a lawyer or an anti-bot engineer, but it will tell you which questions decide whether your idea is cheap, expensive, or a bad idea — and where the honest ranges land. Duskel builds scraping systems and would rather tell you a project is a red light up front than bill you to discover it.
The feasibility checklist
Work through these before anyone quotes you. Each one moves the project up or down a tier, and a good partner will ask all of them in the first conversation rather than after the invoice.
- Is the data actually public? Can an ordinary visitor see it without logging in or paying? Public, visible data is the cleanest case. Anything behind an account, a paywall, or a "members only" wall is a different question entirely — technically and legally — and pushes you toward yellow or red.
- How hard does the site fight back? Look for the tell-tale defences: a Cloudflare or similar challenge page, content that only appears after JavaScript runs, aggressive rate limits, CAPTCHAs, or IP blocks after a few requests. Each of these turns "read the HTML" into "defeat an active adversary," and that's where the engineering hours go.
- Is it behind a login or paywall? Scraping while authenticated means you've almost certainly agreed to terms of service that govern what you can do with the data. That's a legal constraint, not just a technical one, and it's the single fastest way to turn a green project red.
- What do the terms of service and robots.txt say? Check the site's ToS and its robots.txt before you build. Neither is the whole legal picture on its own, but ignoring them is reckless, and both signal how the site owner expects to be treated. A partner who never mentions them is a warning sign.
- What volume and freshness do you actually need? A one-time pull of a few thousand rows is trivial infrastructure. Millions of pages refreshed hourly needs proxies, queuing, and monitoring — a different project with a recurring cost. Be honest about how fresh the data truly needs to be; "real-time" is often "weekly" in disguise, and it's the expensive assumption.
- How likely is the layout to change (schema drift)? Scrapers break when the target redesigns its HTML. A stable, boring site is low-risk; a fast-moving product site will silently break your extraction on a Tuesday. If you need the data continuously, this risk is the whole reason a live scraper is a maintained system, not a finished artifact.
- Is there an API or a licensing option first? The cheapest scrape is the one you don't build. A surprising number of projects exist only because nobody checked for an official API or a data-licensing deal — often cheaper than building the scraper and far cheaper than maintaining it. Check this before anything else.
A word on legality — read this before you build
The responsible framing is simple to state and worth taking seriously: scraping public, non-personal data that a site makes freely visible sits on far safer ground than collecting data behind a login, harvesting personal information, or ignoring a site's stated terms. Terms of service, robots.txt, copyright, and data-protection law (GDPR, CCPA and their equivalents) are all real constraints, and they vary by jurisdiction, by what you collect, and by what you do with it afterwards.
Our posture is boring on purpose: we scrape public data, we respect terms of service and applicable law, we raise the legal picture early rather than pretend it doesn't exist, and for anything that touches personal data or authenticated content we tell clients to get their own legal sign-off. A scraper that collects data you weren't allowed to collect isn't cheap at any price — so if the legal picture is the blocker, that's the answer, and it's better to hear it before you spend.
Green light, yellow, red
Almost every scraping idea sorts into one of three buckets once you've run the checklist. Here's how to read yours.
| Signal | What it looks like | What it means for cost and risk |
|---|---|---|
| 🟢 Green light | Public, visible data. Static or lightly-scripted pages. Modest volume, no real defences, a stable layout, and a clean ToS/robots picture — or an API you can use instead. | A scoped one-off, often from ~$2k. Low legal risk, low maintenance. This is where most projects actually belong. |
| 🟡 Proceed with care | Some anti-bot defences, JavaScript rendering, moderate volume on a schedule, or a layout that changes often. Data is still public but the site clearly prefers you didn't. | A maintained system, not a script — proxies, monitoring, and upkeep. Retainer territory (from ~$3k/mo). Feasible, but price the maintenance in. |
| 🔴 Red flag | Data behind a login or paywall, personal data at scale, terms that explicitly forbid it, or heavy defences you'd have to actively defeat. Or there was an official API all along. | Stop and reassess. Get legal sign-off, use the API, or licence the data. The cheapest scraper here is the one you don't build. |
How to read your scraping project after the checklist
When to hire a specialist — and why pipelines need maintenance
A clean green-light scrape of a cooperative site is often something a capable generalist can handle. The moment your project drifts into yellow — defences, JavaScript rendering, scale, or a continuous feed other systems depend on — a specialist scraping partner usually costs less in the end, because the hard part isn't extracting the data once, it's keeping it flowing when the target actively works against you.
The thing cheap quotes leave out, and the reason continuous scraping is a retainer rather than a one-off, is this: a scraper is not a finished product; it's a thing that decays the moment the target changes its HTML. Sites redesign, add defences, and tighten rate limits without warning. A live pipeline needs monitoring that shouts when the data stops flowing, retry and proxy logic, and someone to fix the extraction when — not if — the layout drifts. A flat one-time price with no plan for upkeep is a scraper that works until the site's next release.
We build scraping systems the way we'd want to depend on them: scoped honestly, with the maintenance priced in rather than pretended away, and with the legal and API questions raised before the build. If you've got a project and want to know which colour it really is — including projects like scraping for sales lead generation where data quality is the whole game — tell us what you're trying to collect and we'll tell you straight whether it's worth building.