Every search engine, every AI system that reads the live web, and every index of any real size starts with the same piece of software: a crawler. It's one of the oldest ideas in how the internet gets organised, and also one of the most misunderstood — mostly because "crawling" gets used loosely to mean almost any kind of automated visit to a website, when it actually describes something specific.
What a crawler does
A web crawler's job is to move through the web by following links, the same way a person would click from page to page, except automatically and at far greater scale. It starts from a set of known pages, reads the links each one contains, and adds the pages it finds to a queue to visit next. Visit those, find more links, add more pages. Repeat, continuously, and the result is a map of a meaningful share of the reachable web — not by anyone deciding in advance what belongs on the map, but by following the web's own structure outward from wherever it starts.
This is a genuinely different job from indexing (organising what was found so it can be searched) or scraping (pulling a specific value off one page). A crawler's whole concern is discovery — what exists, and how to get to it. See crawling, indexing and scraping for how the three fit together.
How a crawler decides where to go
Not every page gets equal attention. A crawler constantly decides what to visit next and how often: pages that change frequently — a news homepage, an active forum — are worth revisiting more than a static reference page untouched for years. A well-designed crawler also has to avoid getting stuck: the web contains genuine infinite loops (calendar pages linking to the next month, forever) and dead ends. Handling a truly dead page sensibly matters too — re-checking something permanently gone wastes effort, which is why a well-run crawl keeps a registry of confirmed-dead pages and lets that status expire after a while, so a page that recovers gets rediscovered rather than staying blacklisted on old information.
What a polite crawler is supposed to do
A crawler running at any real scale touches a lot of other people's servers, and there's a widely respected set of norms — not laws, but strong convention — about how it should behave:
- Read
robots.txtfirst. This file, published by a site itself, states which parts of the site a crawler is and isn't welcome to read, and can specify aCrawl-delay— a minimum gap between requests. A well-behaved crawler treats that delay as a floor, never going faster, and is happy to go slower. - Pace itself. Even without an explicit delay set, hammering a server with requests is a good way to look like an attack rather than a reader. Requests should be spaced out and rate-limited so a crawl is close to unnoticeable in a site's normal traffic.
- Identify itself honestly. A crawler's requests carry a name — a
User-Agentstring — that a site owner can look up, understand, and choose to allow or block. Hiding that identity, or claiming to be an ordinary browser when it isn't, undermines the whole system of consent that makes crawling at scale tolerable. - Stay contactable. A crawler that's misbehaving on someone's site should have a real way for that site owner to reach a human and say so.
askFinz-Crawler is documented publicly against exactly this standard — how it identifies itself, how it paces requests, and how to allow or block it — because a webmaster shouldn't have to guess what an automated visitor is doing on their site.
Why crawlers work at scale at all
A single crawler on one machine hits a hard ceiling — there's only so much of the web one process can read before it's outpaced by how fast the web changes. A crawl that leases out which domain to work on next, atomically, so two machines never duplicate the same work and a dropped machine doesn't strand its share, is how crawling scales past what any one process could manage.
Where crawling leads
On its own, a crawl is just a map — useful, but not yet an answer to anything. What happens after a page is found is where the real value gets created: reading it properly, understanding what it says, and making it searchable. See how a search index is built for the rest of that pipeline, and askFinz's web index for what a continuously crawled and continuously indexed system looks like end to end.
Further reading
- The politeness contract in full, including how to check what's crawling your own site: askFinz-Crawler documentation
- What happens to a page after it's found: How a search index is built
- Why a crawler gets refused sometimes even on pages it's entitled to read: Why a headless browser gets a 403