Every lead-gen scraper demo works. You point it at a directory, it pulls 500 company names and some firstname.lastname@ guesses, and everyone in the room nods. Two weeks later the reps have worked the list: 40% of the emails hard-bounced, a third of the accounts were duplicates or already in the CRM, and the SDR lead has quietly gone back to buying from ZoomInfo. The scrape was never the hard part. The hard part is everything between the raw HTML and a row a salesperson will actually dial without swearing.
We've built pipelines that feed live sales teams, and the split is always the same: the scraper is maybe 20% of the work. The other 80% is enrichment, verification, deduplication, and identity resolution — the plumbing that decides whether the list is worth money or is a spreadsheet of noise with a company name attached. If a vendor pitches "we scrape leads" and stops there, they're quoting you for the 20% and leaving you to discover the rest in your bounce logs.
The scrape is the cheap part; enrichment is the product
A company name and a URL is not a lead. A lead is a named person, a role, a verified way to reach them, and enough context for a rep to write a first line that isn't a mail merge. So the pipeline is stages: discover the company, resolve the real domain (not the LinkedIn URL, not the aggregator's tracking link), find the right people and titles, derive or verify contact details, then attach firmographics — headcount, tech stack, funding round. Every stage drops rows. Start with 10,000 scraped companies and you ship maybe 2,000 leads a rep will touch. That 5:1 attrition is a healthy pipeline, not a broken one.
The common mistake is treating enrichment as one API call to Clearbit or Apollo. Those vendors have coverage holes that fall exactly on the SMB and regional accounts most B2B teams actually want — the 15-to-50-person companies that never filled out a data provider's profile. We run a fallback chain instead: hit the enrichment API, fall back to pattern inference from the domain's known email format, fall back to a targeted scrape of the company's own /about or /team page. That last hop is slow and unglamorous, and it's frequently the only place a 15-person shop's contact data exists at all.
Anti-bot defenses are a freshness tax, not a wall
People assume anti-bot systems stop you from scraping. Mostly they don't — they make each request slow and expensive enough that your data goes stale before you can use it. Cloudflare's JS challenges, datacenter-IP blocks, and Turnstile push you onto residential proxies and headless Chrome, and both cost real cents per page instead of fractions of one. So the engineering question isn't "can we get past it," it's "what's the cheapest tier that returns this specific page." We tier every target: plain HTTP first, then a lightweight fetch through rotating residential IPs, and only then a full headless browser with a warmed fingerprint and human-timed clicks. Routing 100% of traffic through the browser path because 10% of targets demand it is how a pipeline's unit economics quietly go underwater.
The corollary: scrape as little as possible and cache like it costs you money, because it does. A company's headcount doesn't change hourly and its team page doesn't change weekly. We keep a store keyed by normalized domain with a per-field TTL — email deliverability re-checked every few days, firmographics every few months — so a nightly re-run touches a fraction of the targets the first pass did. The teams that burn out on scraping are the ones re-fetching the whole internet every night to catch the 2% of records that moved.
Verify the email or you'll burn the domain
Guessed emails are worse than no emails. Blast a list of firstname.lastname@ guesses and your hard-bounce rate crosses the 3-4% where Google and Outlook start throttling you, your sending domain's reputation drops, and now your genuinely good emails route to spam too. Pattern inference is a fine first guess, but every address has to survive verification before a sequencer sends to it: MX lookup, an SMTP RCPT handshake where the server answers honestly, and catch-all detection so you know when a "valid" result is actually meaningless. Role addresses — info@, sales@, support@ — get flagged and scored separately because they convert worse and some platforms reject them on import.
Catch-all domains are the trap nobody warns you about. Plenty of mail servers accept every address at the handshake and bounce it silently later, so naive verification returns green for a mailbox that goes nowhere. You detect the catch-all, mark those records lower-confidence, and let the rep decide whether to risk them — you don't ship them stamped "verified." Per-field confidence scoring is the honest handoff: a rep working a "90%, catch-all" lead paces it differently than one working an SMTP-confirmed address, and hiding that distinction is how you torch a domain nobody agreed to gamble.
Dedup and identity resolution decide the real yield
The same company shows up as "Acme, Inc.", "Acme Incorporated", and "acme.io" across three sources, and a naive pipeline ships all three as separate rows. Reps hate almost nothing more than dialing an account a teammate closed last month. Real deduplication resolves on the domain as the primary key — strip www, collapse subdomains, fold country-TLD variants like acme.co.uk into acme.com where they're the same entity — and only falls back to fuzzy name matching when no domain exists. Then you dedup against your own CRM and suppression lists, so current customers and open opportunities never resurface as "net-new."
This is also where a list stops being a one-time dump and becomes a system. Companies get acquired, people change jobs every 18 months, domains lapse and get parked. A pipeline worth paying for tracks each record's identity over time and updates it in place, instead of generating a fresh pile of near-duplicates on every run that your ops team then has to merge by hand. That's the line between a scraper and a data asset.
Stay on the legal side of the line
Scraping public business data is broadly defensible, but the line is real and it moves by market. Public firmographics and business contact details are one thing; scraping behind a login (which drags in the site's terms and CFAA-style exposure in the US), or collecting personal data on EU residents without a lawful basis under GDPR, is another. B2B outreach gets a legitimate-interest carve-out in some jurisdictions and runs into hard opt-in rules in others, and "my scraper found it" has never been a defense. We wire suppression and honor-the-opt-out into the pipeline on day one, because bolting compliance on after the first complaint costs far more than designing for it — in fines and in the rebuild. The goal is a list your legal team signs off on, not one you hope nobody asks about.