If you've ever maintained a scraper, you know the pattern: it works fine for weeks, then one morning it returns nothing, or worse, returns garbage that looks fine until someone notices the numbers are wrong. Nothing about your code changed. The website did.
This isn't bad luck or bad engineering. It's the structural problem with scraping as an approach — you're building against a target that has no obligation to stay still.
The selector problem
Most scrapers work by targeting a specific spot in a page's structure: "the price is inside this div, inside this class." That works exactly as long as the page's structure doesn't change. But websites get redesigned constantly — a new checkout flow, a rebranded product page, a framework migration — and every one of those changes has a real chance of moving, renaming, or removing the exact element a scraper was built against.
The scraper doesn't usually fail loudly. It often keeps running and returns something — an empty string, a stale cached value, the wrong field entirely — and nobody notices until a report looks off. Silent failure is the expensive kind.
Sites actively work against it
Beyond ordinary redesigns, a lot of sites deliberately make scraping harder: rate limits that kick in after a handful of requests, content that only appears after JavaScript runs, CAPTCHAs triggered by anything that looks automated, and outright blocks on IP ranges or user agents that don't look like a normal visitor. None of this is necessarily hostile — a site protecting itself from abusive traffic can't always tell the difference between a legitimate reader and a scraper hammering it — but the effect is the same: a pipeline that worked yesterday returns nothing today, and the cause could be any one of a dozen things.
This is a big part of why a headless browser gets blocked while a real one doesn't — sites have gotten good at telling the two apart, and scraping infrastructure usually looks like the former.
Structure isn't the only thing that moves
Even a scraper aimed at a stable page has to deal with content that's dynamic by nature. A price changes by the minute. A news article gets updated after publication. A product goes out of stock. A one-time scrape captures a single moment and calls it current — which it stops being almost immediately. Anyone relying on scraped data for anything time-sensitive is, in effect, always working from a slightly outdated snapshot, and has to keep re-running the same fragile process just to stay slightly less outdated.
What tends to actually hold up
The pattern that survives isn't a smarter scraper — it's a different relationship with the source entirely.
Read for meaning, not for position. A system that understands what a page is saying, rather than which pixel a number sits at, survives a redesign that would break a positional scraper outright. That's the difference between extracting a value and actually indexing a page.
Treat a block as information, not a dead end. A 403 usually just means the specific machine that asked got refused — it doesn't mean the site is unreachable in general. Infrastructure built around that distinction can retry sensibly instead of giving up on a source entirely. A pool-wide registry of pages that are genuinely gone, with entries that expire and get re-tried later, does the same job for pages rather than machines — so a page that recovers becomes reachable again instead of staying blacklisted forever on old information.
Stay current instead of re-running. The alternative to "scrape it again periodically and hope it's still accurate" is reading continuously, so a page is current the moment it's read rather than the moment someone remembers to re-run a job. That's the whole premise behind real-time indexing versus scheduled crawling.
Don't rebuild the wheel per source. Every site that needs a bespoke scraper is another thing that can quietly break. A general-purpose reading system — one built to handle ordinary pages, documents, and the awkward cases without a custom script per site — has far fewer places for a redesign to snap something.
The bigger picture
None of this means scraping is useless — for a narrow, one-off extraction from a page you control or trust to stay stable, it's a reasonable tool. But as the foundation for anything that needs to keep working over months and years, across thousands of sources you don't control, it's the wrong shape for the problem. askFinz's web index is built around reading continuously rather than scraping on a schedule, precisely because the alternative is a maintenance burden that never actually ends.
Further reading
- The specific mechanism behind most scraper blocks: Why a headless browser gets a 403 and a real one doesn't
- What actually replaces scrape-and-hope: Real-time vs scheduled indexing
- The vocabulary worth getting straight first: Crawling, indexing, scraping