robots.txt
An explicit allow rule for our token:
User-agent: askFinz-Crawler
Allow: /Crawler · askFinz-Crawler
WEB /
CRAWLER
For site owners & webmasters
bots@askfinz.ai
If we're on your site, here's everything about it.
askFinz-Crawler reads public and partner pages so they can be found in askFinz search. This page is the documentation for it: how to spot it in your logs, how it behaves while it's there, and who to email about it.
Identify it
What it looks like from your side.
A crawl opens with robots.txt, picks up your sitemap if you publish one, and then works through pages at a steady, spaced pace. Requests carry this User-Agent:
askFinz-Crawler/2.0 (+https://askfinz.com/crawler; search.askfinz.ai)Match on the token askFinz-Crawler rather than the whole string, and on the User-Agent rather than the IP address — the address it arrives from changes.
Your access logexample
waiting…
User-Agent on each of those requests
askFinz-Crawler/1.0 (+https://askfinz.com/crawler; bots@askfinz.ai)Let it in
Allowing askFinz-Crawler.
You don't need to do anything for us to read a public page. These are for the case where something in front of your site is turning us away.
Firewall or WAF
If your firewall filters by user-agent, add askFinz-Crawler to the allow-list so its requests aren't challenged or dropped. Bot-protection products often block unfamiliar agents by default.
The whole family
One name, a few purposes.
Different parts of askFinz read the web for different reasons, and each says which it is. That way you can allow one and refuse another instead of making a single all-or-nothing decision.
| Identifies as | Reads for |
|---|---|
askFinz-Crawler | General web and partner-site reading |
askFinz-News | News and articles |
askFinz-Finance | Company and market reference pages |
askFinz-Research | Scholarly and reference sources |
askFinz-Learn | Courses and study material |
askFinz-Maps | Places and geographic reference |
askFinz-Mentions | Brand and citation tracking |
askFinz-Desktop | The desktop app reading a page you opened |
askFinz-Status | Availability checks — never content |
A few smaller ones exist for specific workspaces and follow the same naming pattern. Every one of them carries a link back to this page in its User-Agent, so you never have to work out who is asking — and bots@askfinz.ai reaches a person if you would rather just ask.
Our commitments
How it behaves on your site.
Follows robots.txt
When it crawls your site it reads your robots.txt first and follows your rules — including Crawl-delay, which it will only ever wait longer than, never shorter.
Paced for your server
Requests are spaced and globally rate-limited, with larger sites crawled no more aggressively than small ones. It is built to be unnoticeable in your load graphs.
Contactable
The crawler carries its own name and links here, where a human answers. If it is doing something you don't want, bots@askfinz.ai reaches us.
Only what's public
It reads pages that are openly reachable. It does not fill in forms, create accounts, or attempt to reach anything behind a login.
Verification
Checking it's really us.
Our requests reach sites through a privacy layer with rotating exits, so askFinz-Crawler does not come from one fixed IP address. Verify by the User-Agent token above rather than by address.
Anyone can put any string in a User-Agent, ours included. If you are seeing traffic claiming to be askFinz-Crawler and behaving badly, that is worth telling us about — it isn't how ours is meant to behave, and we want to know.
If your infrastructure requires IP-based allow-listing, get in touch and we can arrange a dedicated static address for your domain.
The other direction
Would you rather we read you properly?
Verified domain partners are read directly and kept fresh with priority, instead of waiting to be reached in the ordinary course of crawling. If your content is worth finding, this is the way to make sure it is.
Questions
Site owners
ask us these.
- Why does the crawler come from different IP addresses?
- Its connection routes through a privacy layer with rotating exits, so there is no single fixed address to allow-list. Match on the User-Agent instead. If your setup genuinely requires IP allow-listing, contact us and we can arrange a dedicated address.
- Does being crawled cost me anything?
- It shouldn't be noticeable. Requests are spaced out and rate-limited, and your Crawl-delay is always respected. If you're seeing load you can attribute to us, tell us and we'll slow down for your domain.
- Can I get my site indexed faster?
- Yes — verified domain partners are read directly and refreshed with priority rather than waiting to be reached in the normal course of crawling.
- What happens to a page once you've read it?
- It becomes part of the index that askFinz searches when someone asks a question. Answers drawn from your page cite it and link back to you, so the attribution stays with the source rather than being absorbed anonymously.
Part of a bigger system
Where crawled pages go.
The crawler is one way content reaches askFinz. What it reads flows into a real-time index that answers questions with citations back to the sites the answers came from.
Talk to a person
Get in touch.
For anything about how we read your site — slowing down, stopping, removal, or a dedicated address — email bots@askfinz.ai. For partnerships and everything else, use contact.
It's your site
You decide how it gets read.
Being read well should never be something that happens to you. Set the pace, feed your pages in directly with priority and real-time freshness, or ask us not to read the site at all — all three are your call, and none of them requires an account.