"We crawled the site." "We scraped the data." "It's in the index now." In casual conversation these three phrases get swapped for each other constantly, and most of the time nobody notices, because the words are close enough that the sentence still makes sense. But they describe three different jobs, done by three different pieces of software, and mixing them up is the reason a lot of "why isn't my data showing up" confusion happens in the first place.
Here is the plain version, in the order the work actually happens.
Crawling: finding pages
A crawler's job is discovery. It starts from a known page, reads the links on it, and follows them to find more pages — then repeats that process outward, page after page, site after site. A crawler doesn't care what a page says. It cares what it links to. Its output is a map: here is what exists, and here is how it connects.
This is the step that decides coverage. A crawler that only follows a handful of links from a homepage will miss most of a large site. A crawler built to work through a site properly — respecting its robots.txt, pacing its requests so it never looks like an attack — reaches far deeper. What a web crawler actually does goes into this in more detail, and askFinz's own crawler is documented at /crawler for anyone checking their logs.
Scraping: pulling out specific data
Scraping is narrower and more surgical. Instead of asking "what pages exist," scraping asks "what specific values are on this one page" — a price, a phone number, a table of numbers, a list of names. A scraper is usually built against one page's structure: it knows the price sits inside a particular tag, in a particular position, and it pulls that value out.
That precision is also scraping's weakness. It's built against today's version of a page, and a page is not a stable target — see why web scraping breaks for exactly how often, and why. Scraping is a fine tool for a narrow, well-defined extraction job. It is a poor foundation for anything that needs to keep working as the web changes underneath it.
Indexing: making it findable
Indexing is the step that turns "we have a pile of pages" into "you can ask a question and get an answer." It takes what crawling found — and, where documents are involved, what scraping or a similar extraction step pulled out — and organises it so it can be searched instantly, rather than re-read from scratch on every query.
A good index stores more than a keyword list. Modern indexes — including askFinz's own — store the meaning of a passage, not just the words in it, so a question can be matched against ideas rather than a literal string. That's what separates "we have a database of pages" from "you can ask this in plain English and get an answer."
Why the distinction actually matters
The confusion between these three terms isn't just pedantic. It changes what you should expect from a tool:
- If someone says they "crawled" a site, ask whether they found the whole thing or just the homepage and a few links deep.
- If someone says they "scraped" a site, ask what happens when that site changes its layout next month — because it will.
- If someone says a site is "indexed," ask whether that means it's stored as a link, or actually read and searchable by meaning.
A search product that only crawls has coverage without content. One that only scrapes has content without durability. One that only indexes without a live source behind it goes stale. The three have to work together — crawling to find, extraction to read, indexing to make it usable — and the order they run in is the order listed above, every time.
Where this fits together
Most of the friction people run into with "the data isn't there" or "the answer is out of date" traces back to one of these three steps being weaker than the others. A crawler that can't get past a login wall means indexing never gets a chance. A scraper tied to last year's page layout produces indexed garbage. An index that's rebuilt on a schedule rather than continuously means real crawling and scraping happened, but the results sat unused for days.
Seeing how the pieces connect end to end is easier with the full picture: how a search index is built walks through the whole pipeline, and how the askFinz index works shows what it looks like when crawling and indexing run together, continuously, rather than as separate scheduled jobs.
Further reading
- What a web crawler actually does, and how to be polite to one: What is a web crawler?
- Why scraped data goes stale: Why web scraping breaks
- The full pipeline from a URL to an answer: How a search index is built