Crawl. Read.Store. Retrieve.That's web indexing.
Web indexing is the process of finding pages, reading them, keeping what matters, and making it findable again. Every search engine does some version of it. What differs — and what this page is really about — is what gets kept, and what an answer can be built from.
web indexing/
├── crawl/
│ └── find the pages
├── read/
│ └── pull out what they say
├── store/
│ ├── links only classic
│ └── meaning askfinz
└── retrieve/
├── a list of pages classic
└── the passage askfinzFour steps, and nothing more mysterious.
Strip away the branding and every web index — ours included — does the same four things. Where products diverge is what happens at step three.
Crawl
Something goes and finds pages — following links, reading a sitemap, or being handed a page directly. This is the part most people picture when they hear "crawler."
Read
The page is opened and its content pulled out — the article, the text, the tables — separated from the navigation and the clutter around it.
Store
What was read is kept somewhere, organised so it can be found again. This is the index itself — a library, not a to-do list of pages left to visit.
Retrieve
When someone asks a question, the store is searched and the relevant part is handed back — ideally in the time it takes to notice a page loaded at all.
Classically, a map from words to pages.
A traditional search index is built around one question: which pages contain these words? It keeps a table from every word it has seen to the pages that used it, then ranks the matches by signals like how many other pages link to them. That is why a search for the right words works well and a search for the right idea, phrased differently, often doesn't — the index never stored the idea, only the words.
A classic, link-based index
- Built to answer "which pages contain these words" — a lookup table from words to pages.
- Ranks by signals like links and popularity, because it has no way to judge whether a page actually answers the question.
- Hands back a list of pages and leaves the reading to you.
A meaning-based index — what askFinz keeps
- Built to answer "what does the web say about this" — reads a page for what it means, not which words it used.
- Ranks by whether a passage answers the question, because it kept the passage, not just a pointer to the page it lived on.
- Hands back the passage itself, already matched to what was asked.
The kind of index you build sets the ceiling on the answer.
An index that only stores links can only ever hand back links — the reading is left to whoever asked. An index that stores meaning can hand back the answer itself, with a citation to where it came from. That difference is not a matter of degree; it decides what the product built on top of it is even capable of doing.
Live, from the running index
query → vector on the GPU pool · payload-indexed SafeSearch filter · deterministic result cache
Is web indexing the same as web crawling?
Crawling is one step inside indexing — the part that finds pages. Indexing is the whole pipeline: crawl, read, store and retrieve. A crawler with nowhere to put what it finds isn't an index; it's a list of visited addresses.
How do internet search indexes normally work?
Classically, by building a map from words to the pages that contain them, then ranking matches by signals like how many other pages link to them. It's fast and it scales, but it can only ever tell you which pages used the right words — not which ones actually answer the question.
Why does it matter which kind of index a product uses?
Because it decides what an answer is allowed to be. A link index can hand you ten pages to go read yourself. An index built on meaning can hand you the actual passage that answers the question, cited back to where it came from — a materially different thing to build a product on.
Does askFinz run its own index?
Yes. Most AI products buy their web results from someone else's index; askFinz reads the web itself and keeps what it finds. See how it's built on how-the-index-works.
An index built on meaning, not links.
Everything above is easier to believe once you've seen it answer something. Try a real question, or read how the pipeline behind it actually runs.