FyndableBot

• Your robots.txt and sitemap, then up to 50 pages, starting from the home page.
• Web pages only: it skips images and other files.
• Its requests to your site start at least 1 second apart, and it stops after a couple of minutes.
• It doesn't log in, fill in forms or change anything on your site.
• Fyndable's speed test of your home page is fetched by Google PageSpeed Insights, not by FyndableBot.

FyndableBot follows your robots.txt: the rules for FyndableBot, or the rules for all crawlers when there are none for it. To keep it off your whole site, add this:

User-agent: FyndableBot
Disallow: /

Fyndable can't check a site it isn't allowed to read, and tells the person who added it.

Bot protection (Cloudflare's, a firewall or a security plugin) can turn FyndableBot away even when robots.txt lets it in. Fyndable won't try to get past it: FyndableBot never poses as a browser, never uses another user agent and never goes through proxies to look like someone else. To let it in, make an exception for requests whose user agent contains FyndableBot.

FyndableBot doesn't come from a fixed list of IP addresses, so allow it by its user agent rather than by IP. It only ever reads what any visitor to your site can see.

On Cloudflare

1. Sign in to Cloudflare and open your website.
2. Go to Security, then Settings, and filter by Bot traffic.
3. If Bot fight mode is on, turn it off. It can't make an exception for one crawler, so on Cloudflare's free plan this is the only way to let us in.
4. If you have Super Bot Fight Mode (paid plans) and want to keep it on, add an exception instead: go to Security, then Security rules, and choose Create rule, then Custom rules. Choose Edit expression and paste the expression below, set the action to Skip with All Super Bot Fight Mode rules ticked, place the rule first, and select Deploy.
5. Still turned away? In Security, then Analytics, the Events tab lists requests from FyndableBot: its Service column says which setting stopped them.

Expression

(http.user_agent contains "FyndableBot")

Copy

On other bot protection or firewalls

1. Find what protects your website from bots: a service like Cloudflare, a firewall in your hosting control panel, a security plugin, or a setting in your website builder.
2. Add an allow rule (an exception, or allowlist entry) for visitors whose user agent contains FyndableBot.
3. If it can only allow IP addresses, ask its support how to allow one crawler by its user agent: our crawler doesn't use fixed IP addresses.

User agent contains

FyndableBot

Copy

Its requests carry this user agent: