Here's a scenario that trips up a lot of engineers the first time they hit it: you write a script using headless Chromium, point it at a page you can open fine in your normal browser, and it comes back with a 403. Same URL. Same cookies, if you copy them over. Sometimes even the same IP address. And yet one gets the page and the other gets refused outright.
The instinct is to assume something about the request is wrong — a missing header, a bad cookie, the wrong referrer. Usually none of that is it. The real answer is blunter: the site fingerprinted the browser itself, decided it was automation, and refused before your request was ever really considered on its merits.
What actually gets checked
A "headless browser" is a real browser engine — Chromium, usually — running without a visible window, driven by code instead of a person clicking around. It executes JavaScript, renders pages, and can look almost identical to an ordinary visit at the network level. The catch is "almost." Sites that care about bot traffic don't just look at the request; they look at dozens of small signals about how the browser is behaving — timing patterns, how it responds to certain scripts, properties of the browser environment that differ subtly in an automated context versus a human-driven one. Put together, those signals are a fingerprint, and a well-built one is hard to fake convincingly.
This is measurable, not theoretical. On one paywalled site, tested on the same machine with the same cookies: a plain HTTP fetch that simply carried the cookies along returned the full article — a real page, around 7.7 million characters of content. Headless Chromium, hitting the exact same URL, got a 403 with 657 characters back. The response was byte-identical whether or not the cookies were even included — which is the tell. The site wasn't checking whether the visitor was logged in. It had already decided, from the shape of the browser itself, that the session wasn't worth evaluating at all.
Why this happens before anything else
The important detail is the ordering. It's tempting to think of a block as the last line of defence — a check that happens after the site has looked at what you're asking for and decided you're not welcome. In practice it's usually the first thing that happens. The fingerprint check runs, fails, and the request never gets far enough to be evaluated on identity, permissions, or content. A 403 in this context means "this specific browser session was refused," not "this content is unreachable" or "this site has no public pages." Those are very different problems with very different fixes.
What doesn't fix it
A lot of the usual advice — rotate your headers, spoof a user-agent string, add a random delay between requests — treats the symptom rather than the cause. Sites that fingerprint seriously aren't reading your User-Agent header and taking it at its word; they're watching how the browser actually behaves once it's running. Cosmetic changes to a headless script don't change that behaviour, which is why so many scraping setups work for a while and then quietly stop working the moment a site tightens its detection.
What actually works
A real browser gets the page. That's the whole answer, unglamorous as it is. If the thing making the request behaves like an ordinary browser because it is one — not simulated, not stripped down, not running in a mode that leaves a fingerprint behind — there's no automation signature to detect in the first place. This is a meaningful part of why a reading system built around real browser sessions reaches sites that a headless-scraping setup gets locked out of, and it's one reason askFinz's web index reads from many different surfaces rather than relying on one central scraping process trying to look like everyone at once.
It's also why blocks shouldn't be treated as permanent verdicts about a site. A refused session today doesn't mean the site's pages are gone or unreadable — it means that specific attempt didn't pass a fingerprint check, and a different approach might. Infrastructure that distinguishes "this page is truly dead" from "this attempt was refused" ends up reaching far more of the web than one that treats every 403 the same way.
Further reading
- The broader pattern behind fragile automated access to the web: Why web scraping breaks
- How a crawler is meant to identify itself and behave politely instead of hiding: What is a web crawler?
- The full account of how askFinz reads the web without relying on one blockable path: How the index works