A scraping API has to answer at 9am when the dashboard loads, not whenever the target site feels cooperative. Here is the architecture that holds: async jobs, versioned schemas, freshness as a parameter, and block detection that treats a 200 as a lie.
The proxy bill landed before the data did. A client's dashboard fired 1,200 profile lookups at a Flask route wrapping a script that had run fine on someone's laptop for a year. The target rendered behind a JS challenge in 41 seconds; the load balancer cut the connection at 30; the frontend retried three times. Four residential proxy sessions burned to produce one gateway timeout, times 1,200 rows nobody ever saw. No selector was broken. The architecture was wrong the moment a human started waiting at the other end of the socket.
The difference between a scraper and a scraping API is who absorbs the uncertainty. A script gets to fail at 3am and be re-run by the person who wrote it. An API is a promise to a caller who does not know the target site exists and who will hang a checkout flow, a lead score, or a repricing rule off your response body. We keep extraction running across roughly 300 domains for commerce, recruiting and market-data clients, and every scar comes from the same seam: the messy web on one side, the clean contract we sold on the other.
Stop pretending the fetch is synchronous. POST /v1/jobs validates the URL or query, writes a row, returns 202 with a job ID in about 40 milliseconds, and drops the work on a per-domain queue. The caller polls GET /v1/jobs/{id} or hands you a webhook. A slow target becomes a job that takes 90 seconds instead of a request that dies at 30, and your p99 stops being a function of somebody else's Cloudflare settings. Workers can then do the sensible things — token bucket per domain, batching, reordering around a site's quiet hours — none of which are possible while a socket is held open.
Two details decide whether this holds. Idempotency keys first: hash customer ID, canonical URL and schema version into a Redis key with a 24-hour TTL, so a caller retrying your own 502 gets the same job ID back rather than a second fetch. One retry-happy client burned about $2,800 of residential proxy credit in an afternoon that way; the fix was fifteen lines. Second, jobs need a state machine — queued, fetching, rendering, extracting, done, blocked, failed — not a boolean. A blocked job lets the caller choose: escalate to a browser fetch and pay 30x, or serve this morning's cached row. Collapse it all into a 500 and every caller invents its own wrong guess.
You sell fields, not pages. That means a typed, versioned output schema that outlives the target's redesign, which lands the week your client demos to their board. Define it first: price as an integer in minor units plus an ISO 4217 code, never the string "$1,299.00"; availability as a five-value enum, never whichever noun the merchandiser picked this quarter; nullable is declared in the schema or it is a bug. Extraction maps HTML into that schema, and validation runs before a single row is written. Selectors live in versioned per-domain config, shippable in a minute without deploying the service.
Then be honest about partial rows. If nine of twelve fields extracted, do not return three nulls and let the client's pricing model read missing as zero — mark the row partial and name the failed fields. Rows that fail validation go to a quarantine table with the gzipped HTML snapshot that produced them. Fixing a selector takes ten minutes when you can read the exact bytes that broke, and half a day when the page has changed twice since. At a couple of cents per gigabyte-month, snapshots are the cheapest debugging tool in the stack.
Treating every request as a live fetch is the most expensive habit in this business. A plain GET through a datacenter proxy costs a small fraction of a cent. The same page through a headless browser on a residential IP costs thirty to sixty times that once bandwidth, CPU and retries are counted. So let callers state their tolerance: max_age in seconds, checked against a cache keyed on canonical URL plus schema version. A rep opening a company profile is fine with data from this morning. A repricing engine at 11pm is not. Same endpoint, two orders of magnitude of cost between them, and only the caller knows which one they are.
Underneath, tier the fetch per domain rather than globally. Try a plain request with a coherent header set and TLS fingerprint first; fall back to a real browser only when the response fails a content check. On one commerce client, moving from browser-everything to HTTP-first took monthly infrastructure from about $4,100 to $700 and raised the success rate at the same time, because fewer concurrent browsers meant less CPU contention and fewer self-inflicted timeouts. Track fallback rate per domain as a first-class metric: a domain jumping from 8% to 60% overnight is an incident, not a line item.
Anti-bot systems rarely hand you a clean 403 anymore. You get a 200 with an interstitial, or the real layout with prices stripped out, or a geo variant priced in another currency, or an A/B bucket where the DOM you built against no longer exists. If your health check is the HTTP status, you will write thousands of empty rows into a client's warehouse and hear about it when their analyst asks why every competitor went free on Tuesday. Define success on content: required fields present, values inside plausible ranges, response body within a band of the rolling median size for that domain.
The mechanism that catches drift early is a canary set — twenty to fifty URLs per domain with known, slow-moving values, fetched on their own schedule outside the customer job flow. When canary extraction falls below its rolling baseline, the domain is flagged and someone opens a stored snapshot before the queue writes another batch. Alert on the boring statistics too: a field's null rate doubling day over day, median price moving past a threshold, row counts falling off a cliff. Each of those has caught a silent break before the client saw it, which is the whole product. Nobody buys a scraping API for the happy path. They buy it so the pager that fires when the site changes is ours.
A software studio that ships and maintains its own products — KeepChats, Gwora and MoveProof — and builds the same way for clients. Founded and led by codewithumar.
Talk to the studio →We build software worth keeping — for clients, and for ourselves.
Founded & led by codewithumar