Skip to content
Metricsjar

Documentation · Updated August 2026

Allow Metricsjar Site Audits Through Cloudflare

Metricsjar visits your public pages as an automated crawler. Cloudflare can challenge that request even when the same page opens normally in your browser. Metricsjar now signs every crawl request with Web Bot Auth, so Cloudflare can verify the crawler without relying on a fixed IP address.

When this happens, Metricsjar stops the audit. It does not treat Cloudflare's challenge page as your website or report the challenge page's noindex tag as an SEO problem.

What Web Bot Auth changes

Each request includes a short-lived Ed25519 HTTP message signature and points to Metricsjar's signed public key directory. Cloudflare verifies that signature before classifying the request. The signature covers the destination hostname, expires after one minute, and is generated again for every HTML, asset, and Lighthouse request.

Once Cloudflare lists Metricsjar as a verified bot, its known-bot field cf.client.bot is true for a valid signed request. This lets you keep bot protection enabled while allowing verified SEO crawlers. A valid signature identifies Metricsjar; it does not override your WAF rules or your decision to block verified bots.

Confirm what Cloudflare blocked

Do this before changing a security rule:

  1. Open your domain in the Cloudflare dashboard.
  2. Open Analytics → Events. In the previous dashboard, open Security → Events.
  3. Set a short time window around the audit.
  4. Filter for the Ray ID shown by Metricsjar. If no Ray ID was returned, filter for the user-agent MetricsjarSiteAudit/2.0 (+https://metricsjar.com).
  5. Expand the matching event and read its Service and Rule fields.

Cloudflare recommends Security Events as the starting point for false positives. A Ray ID connects the response Metricsjar received to the security event that produced it. Security Events can be sampled, so use the shortest available time window if the event does not appear.

Allow verified bots in Super Bot Fight Mode

Super Bot Fight Mode has a built-in verified-bot policy:

  1. Open Security → Settings.
  2. Filter by Bot traffic.
  3. Set Verified bots to Allow.
  4. Run the audit again.

This is the preferred recovery. It uses Cloudflare's cryptographic classification instead of trusting a user-agent or a hosting-provider IP range.

If a custom WAF rule caused the block

Custom rules run before Super Bot Fight Mode. A terminating Block or Challenge action can therefore stop a verified crawler before the built-in bot policy is evaluated.

Cloudflare's canonical exception for known bots is:

(cf.client.bot)

Create a Skip rule for that expression, select All remaining custom rules, and place it first. This allows all Cloudflare verified bots, not only Metricsjar, so review whether that is appropriate for the protected paths on your site. You can instead add and not cf.client.bot to the specific Block or Challenge rule that fired.

If the service is Bot Fight Mode

Cloudflare's free Bot Fight Mode does not support WAF Skip rules. Web Bot Auth still gives Cloudflare a cryptographic crawler identity, but there is no per-crawler override if Bot Fight Mode itself continues to challenge the request.

  1. Open Security → Settings and filter by Bot traffic.
  2. Turn Bot fight mode off.
  3. Run the Metricsjar audit again.
  4. Turn Bot Fight Mode back on after the audit.

For a permanent verified-bot policy, move to Super Bot Fight Mode or Bot Management. Do not add a broad IP, ASN, or user-agent exception: those signals are shared or copyable, while Web Bot Auth proves possession of Metricsjar's private key.

If the event shows an origin response instead of a Cloudflare security action, check your hosting firewall, security plugin, and origin access logs for the same timestamp.

Why robots.txt does not fix this

robots.txt tells a cooperative crawler which URLs it may fetch. It does not override Cloudflare's WAF or bot protection. Cloudflare can challenge a request before it reaches your origin or reads your site's crawler policy.

Metricsjar respects robots.txt after it can reach the site. A Web Bot Auth signature identifies the crawler but does not grant permission to ignore your crawl policy.

A geographic block is different

Some websites permit visitors only from specific countries or regions. That is not a bot-protection false positive, and changing a Cloudflare bot rule will not fix it.

The default Auto crawl location starts in North America. If Metricsjar receives a geographic access page, it retries once from Europe before running the SEO checks. You can also choose North America, Europe, or Asia-Pacific in the audit form. Every completed report records the location that served the crawl, including whether Auto used the European fallback.

If the site still returns a geographic access page, choose a location where the site is intended to operate. Metricsjar stops instead of auditing the access page as if it were the website.

How to confirm Web Bot Auth was sent

When Metricsjar receives a Cloudflare block through its regional gateway, the paused audit shows Web Bot Auth identity sent. This confirms that the gateway signed the request. It does not by itself prove that Cloudflare accepted the signature or allowed the request.

Cloudflare records an accepted, registered signature as a known bot. If the event still shows cf.client.bot as false, use the Ray ID from the Metricsjar result when contacting Metricsjar support; it distinguishes a verification problem from a later WAF decision.

Primary references

Keep reading

Documentation

Get Started With Metricsjar

Documentation

Read Your Dashboard