For site owners & webmasters
bots@askfinz.ai
The askFinz search crawler. Named, slow, signed, and it takes no for an answer.
askFinzBot reads public pages so people can find them through askFinz search, always with a link back to the source. It follows your robots.txt, keeps a gentle pace, signs every request so you can check it is really us, and your pages are never used to train AI models.
Every request carries this User-Agent, and it never changes from one request to the next:
Mozilla/5.0 (compatible; askFinzBot/1.0; +https://askfinz.com/crawler)In robots.txt and firewall rules, match on the name askFinzBot. Its network address can change, so do not allow or block it by IP. Because anyone can copy a User-Agent, the signature described below is the real proof that a request is ours.
Before it asks for any page it reads your robots.txt and follows the rules written for askFinzBot, or for everyone if there are none. If your robots.txt cannot be reached because your server is having trouble, it does not crawl at all.
Never more than one request to your site at once, at least 10 seconds apart, and at most 500 pages a day. A Crawl-delay you set only ever makes it slower. If your server answers "too many requests" or "busy", it backs off.
Each request carries a cryptographic signature that proves it came from askFinz, so you and your security provider can tell the real askFinzBot from anything pretending to be it.
It does not solve CAPTCHAs or try to get around bot checks. If your site still turns it away, it stops and does not try again by other means.
Add a group to your robots.txt. askFinzBot reads it again at least every hour, so a change takes effect within the hour.
User-agent: askFinzBot
Disallow: /User-agent: askFinzBot
Disallow: /private/
Disallow: /drafts/User-agent: askFinzBot
Crawl-delay: 30A group for askFinz also applies to askFinzBot. With no group naming it, it follows the rules you set for all crawlers (User-agent: *).
askFinzBot signs every request using Web Bot Auth, an open standard built on HTTP Message Signatures. Each request carries three headers: Signature-Agent, Signature-Input and Signature. The signature is made with a key that never leaves our own servers and is only valid for about a minute.
Our public key is published in our key directory. Security services that support Web Bot Auth fetch it and check the signature for you.
https://askfinz.com/.well-known/http-message-signatures-directoryA request that names askFinzBot but carries no valid signature is not from us. We would like to hear about it at bots@askfinz.ai.
What askFinzBot reads goes into the askFinz search index. When an answer draws on your page, it names your page and links to it, so readers can go straight to the source.
Pages askFinzBot reads are marked as they are stored and are kept out of every data set we use to train models. They are used for search and reference only.
Besides askFinzBot, parts of askFinz read the web under their own names. Everything below is about those readers, not about askFinzBot.
A crawl opens with robots.txt, picks up your sitemap if you publish one, and then works through pages at a steady, spaced pace. Requests carry this User-Agent:
askFinz-Crawler/2.0 (+https://askfinz.com/crawler; search.askfinz.ai)Match on the token askFinz-Crawler rather than the whole string, and on the User-Agent rather than the IP address — the address it arrives from changes.
askFinz-Crawler/1.0 (+https://askfinz.com/crawler; bots@askfinz.ai)You don't need to do anything for us to read a public page. These are for the case where something in front of your site is turning us away.
An explicit allow rule for our token:
User-agent: askFinz-Crawler
Allow: /If your firewall filters by user-agent, add askFinz-Crawler to the allow-list so its requests aren't challenged or dropped. Bot-protection products often block unfamiliar agents by default.
Different parts of askFinz read the web for different reasons, and each says which it is. That way you can allow one and refuse another instead of making a single all-or-nothing decision.
| Identifies as | Reads for |
|---|---|
askFinz-Crawler | General web and partner-site reading |
askFinz-News | News and articles |
askFinz-Finance | Company and market reference pages |
askFinz-Research | Scholarly and reference sources |
askFinz-Learn | Courses and study material |
askFinz-Maps | Places and geographic reference |
askFinz-Mentions | Brand and citation tracking |
askFinz-Desktop | The desktop app reading a page you opened |
askFinz-Status | Availability checks — never content |
A few smaller ones exist for specific workspaces and follow the same naming pattern. Every one of them carries a link back to this page in its User-Agent, so you never have to work out who is asking — and bots@askfinz.ai reaches a person if you would rather just ask.
When it crawls your site it reads your robots.txt first and follows your rules — including Crawl-delay, which it will only ever wait longer than, never shorter.
Requests are spaced and globally rate-limited, with larger sites crawled no more aggressively than small ones. It is built to be unnoticeable in your load graphs.
The crawler carries its own name and links here, where a human answers. If it is doing something you don't want, bots@askfinz.ai reaches us.
It reads pages that are openly reachable. It does not fill in forms, create accounts, or attempt to reach anything behind a login.
Our requests reach sites through a privacy layer with rotating exits, so askFinz-Crawler does not come from one fixed IP address. Verify by the User-Agent token above rather than by address.
Anyone can put any string in a User-Agent, ours included. If you are seeing traffic claiming to be askFinz-Crawler and behaving badly, that is worth telling us about — it isn't how ours is meant to behave, and we want to know.
If your infrastructure requires IP-based allow-listing, get in touch and we can arrange a dedicated static address for your domain.
Verified domain partners are read directly and kept fresh with priority, instead of waiting to be reached in the ordinary course of crawling. If your content is worth finding, this is the way to make sure it is.
The crawler is one way content reaches askFinz. What it reads flows into a real-time index that answers questions with citations back to the sites the answers came from.
For anything about how we read your site — slowing down, stopping, removal, or a dedicated address — email bots@askfinz.ai. For partnerships and everything else, use contact.
Being read well should never be something that happens to you. Set the pace, feed your pages in directly with priority and real-time freshness, or ask us not to read the site at all — all three are your call, and none of them requires an account.