Before you commission a scraper, run this checklist: is the data public, how hard does the site fight back, and what does the law say? Here's how to tell a green-light project from a money pit.
Most of a web scraping project's cost and risk is decided before a single line of code is written — by the target site, not the data you want off it. A scrape can be a two-hour job or a permanent engineering commitment, and the difference is almost never obvious from looking at the page. So before you build one or pay someone to, it's worth spending twenty minutes deciding whether the project is a green light, a proceed-with-care yellow, or a red flag you should walk away from.
This is the checklist we run on every scraping enquiry before we quote it. It won't make you a lawyer or an anti-bot engineer, but it will tell you which questions decide whether your idea is cheap, expensive, or a bad idea — and where the honest ranges land. Duskel builds scraping systems and would rather tell you a project is a red light up front than bill you to discover it.
A short version: the more of these answers land on "yes, and the site doesn't want me here," the more the project shifts from a one-off script toward an ongoing, maintained data feed — and the more the legal picture matters.
Work through these before anyone quotes you. Each one moves the project up or down a tier, and a good partner will ask all of them in the first conversation rather than after the invoice.
This is not legal advice, and nothing here is a substitute for a lawyer who knows your jurisdiction and use case. What follows is how we think about it as engineers, not a legal opinion.
The responsible framing is simple to state and worth taking seriously: scraping public, non-personal data that a site makes freely visible sits on far safer ground than collecting data behind a login, harvesting personal information, or ignoring a site's stated terms. Terms of service, robots.txt, copyright, and data-protection law (GDPR, CCPA and their equivalents) are all real constraints, and they vary by jurisdiction, by what you collect, and by what you do with it afterwards.
Our posture is boring on purpose: we scrape public data, we respect terms of service and applicable law, we raise the legal picture early rather than pretend it doesn't exist, and for anything that touches personal data or authenticated content we tell clients to get their own legal sign-off. A scraper that collects data you weren't allowed to collect isn't cheap at any price — so if the legal picture is the blocker, that's the answer, and it's better to hear it before you spend.
Almost every scraping idea sorts into one of three buckets once you've run the checklist. Here's how to read yours.
| Signal | What it looks like | What it means for cost and risk |
|---|---|---|
| 🟢 Green light | Public, visible data. Static or lightly-scripted pages. Modest volume, no real defences, a stable layout, and a clean ToS/robots picture — or an API you can use instead. | A scoped one-off, often from ~$2k. Low legal risk, low maintenance. This is where most projects actually belong. |
| 🟡 Proceed with care | Some anti-bot defences, JavaScript rendering, moderate volume on a schedule, or a layout that changes often. Data is still public but the site clearly prefers you didn't. | A maintained system, not a script — proxies, monitoring, and upkeep. Retainer territory (from ~$3k/mo). Feasible, but price the maintenance in. |
| 🔴 Red flag | Data behind a login or paywall, personal data at scale, terms that explicitly forbid it, or heavy defences you'd have to actively defeat. Or there was an official API all along. | Stop and reassess. Get legal sign-off, use the API, or licence the data. The cheapest scraper here is the one you don't build. |
Most projects aren't purely one colour — a green data source with a yellow freshness requirement, say. The point of the checklist is to know which parts are which before you commit, so nothing expensive or risky surprises you later.
A clean green-light scrape of a cooperative site is often something a capable generalist can handle. The moment your project drifts into yellow — defences, JavaScript rendering, scale, or a continuous feed other systems depend on — a specialist scraping partner usually costs less in the end, because the hard part isn't extracting the data once, it's keeping it flowing when the target actively works against you.
The thing cheap quotes leave out, and the reason continuous scraping is a retainer rather than a one-off, is this: a scraper is not a finished product; it's a thing that decays the moment the target changes its HTML. Sites redesign, add defences, and tighten rate limits without warning. A live pipeline needs monitoring that shouts when the data stops flowing, retry and proxy logic, and someone to fix the extraction when — not if — the layout drifts. A flat one-time price with no plan for upkeep is a scraper that works until the site's next release.
We build scraping systems the way we'd want to depend on them: scoped honestly, with the maintenance priced in rather than pretended away, and with the legal and API questions raised before the build. If you've got a project and want to know which colour it really is — including projects like scraping for sales lead generation where data quality is the whole game — tell us what you're trying to collect and we'll tell you straight whether it's worth building.
This isn't legal advice, but the honest framing is: scraping public, non-personal data that a site makes freely visible sits on far safer ground than collecting data behind a login, harvesting personal information, or ignoring a site's terms of service. Terms of service, robots.txt, copyright, and data-protection law like GDPR and CCPA are all real constraints that vary by jurisdiction and use case. For anything touching personal or authenticated data, get your own legal sign-off before you build.
Look for the defences: a Cloudflare or similar challenge page, content that only appears after JavaScript runs, CAPTCHAs, aggressive rate limits, or IP blocks after a few requests. A static page with visible, structured data is easy. A site with those defences, high volume needs, or a fast-changing layout is a maintained engineering system, not a quick script — and it should be priced as one.
Always, and first. A lot of scraping projects exist only because nobody looked for an official API or a data-licensing option — which is often cheaper than building the scraper and far cheaper than maintaining it. The cheapest, lowest-risk scrape is the one you never have to build.
Because a scraper isn't a finished product — it's tightly coupled to the target site's HTML, which changes without warning. Sites redesign, add anti-bot defences, and tighten rate limits, and any of those can silently break your extraction. A continuous data feed needs monitoring, retry and proxy logic, and someone to fix it when the layout drifts, which is why ongoing scraping is a retainer rather than a one-off build.
A clean, public, static-site scrape is fine for a capable generalist. Hire a specialist once the project involves anti-bot defences, JavaScript rendering, high volume, tight data-quality requirements, or a live feed other systems depend on — that's where generalists lose weeks and where a silently-broken pipeline costs far more than the rate saved.
A software studio that ships and maintains its own products — KeepChats, Gwora and Cairn — and builds the same way for clients. Founded and led by codewithumar.
Talk to the studio →Send the problem. You get one fixed number and a plan back within a business day.
We build software worth keeping — for clients, and for ourselves.
Founded & led by codewithumar