robots.txt
Add an explicit allow rule for our user-agent token:
User-agent: askFinz_Crawler
Allow: /Crawler · askFinz_Crawler
WEB /
CRAWLER
For site owners & webmasters
Meet askFinz_Crawler.
askFinz_Crawler visits partner and public web pages so they can be found in askFinz search. It obeys robots.txt, identifies itself on every request, and stays rate-limited so it's never a burden on your infrastructure.
What it does
One job: build a good index.
askFinz_Crawler reads partner and public pages across the web so they can be surfaced in askFinz search. It doesn't impersonate a browser, doesn't hide its intent, and doesn't crawl faster than your site can comfortably handle. If you'd rather it not visit at all, that's respected too — see below.
Identify our crawler
Every request carries this User-Agent.
Look for this exact string in your server or access logs to spot askFinz_Crawler:
askFinz_Crawler/1.0 (+https://askfinz.com/crawler)Let it in
Allowing askFinz_Crawler.
Firewall / WAF
If your firewall or WAF filters bots by user-agent, add askFinz_Crawler to your allow-list so its requests aren't challenged or dropped.
Prefer not to be indexed
Blocking askFinz_Crawler.
Add a disallow rule to your robots.txt and we'll honor it on our next crawl:
User-agent: askFinz_Crawler
Disallow: /Once this is live, askFinz_Crawler stops requesting your pages starting with its next visit.
Our promise
How it behaves on your site.
Respects robots.txt
Before it requests anything else, it fetches your robots.txt and follows your rules, including crawl-delay.
Identifies itself
Every request carries the askFinz_Crawler User-Agent below — no disguising as a browser.
Rate-limited
Requests are throttled so the crawler stays polite and doesn't add meaningful load to your site.
Only what you expose
It indexes public and partner pages — never content behind a login it can't reach on its own.
Verification
Verifying it's really us.
askFinz_Crawler currently reaches sites through a rotating VPN egress, so it does not come from a single fixed IP address. Verify its requests by the askFinz_Crawler User-Agent above rather than by IP.
If you require IP-based allow-listing, contact us and we can arrange a dedicated static address.
Part of a bigger system
Where crawled pages go.
The crawler is one way content reaches askFinz. Everything it reads flows into a real-time index that powers search, research, chat, and news — and for pages that need a real browser to render, they're handled automatically.
Questions
Get in touch.
For partnership requests or questions about how we crawl your site, reach us through contact.