When a Scan Can't See the Page, It Should Say So

If a client-side security monitoring tool shows a green check next to your checkout page, you assume it looked at the checkout. Sometimes it never did. Bot protection may have shown the scanner a challenge page, or the page did not load at all. Reporting a page as fine without seeing it is called a false negative, and it is worse than a false alarm, because nobody investigates a page that looks healthy.

Challenge pages

Bot protection services such as Cloudflare sometimes answer an automated visitor with a challenge page, which asks the visitor to prove it is human. Many sites run this protection as part of a web application firewall (WAF), a service that filters visitors before they reach the site. To a program, some challenge pages look like any other web page, with scripts of their own, so a scanner that only checks whether the request succeeded can end up storing the challenge page as if it were the checkout.

We recognize the challenge and block pages of the bot protection services we know and mark the scan as blocked, so a scan that hits one of them can't produce a clean result. We never use a blocked scan as the reference for later scans. A protection page we haven't seen yet could still pass for the real page. We add new ones as we find them, and the check at the end of this article shows how to spot one yourself.

You can allow our scanner through your firewall. If you do, and a challenge page shows up again later, we mark the rule as no longer working and let you know.

When a check fails

Checks sometimes fail. A request times out, or a file arrives incomplete. It is tempting to read a failed check as a negative answer, like “no firewall” or “nothing suspicious”. We record it as not measured:

• A changed script that we could not read in full doesn't count as checked. Once a site is past its first week of monitoring, that alone sends the owner an email.
• If the firewall check ends with an error, it does not report that there is no firewall, and the scan switches to the method we use for protected sites.
• A scan that failed is shown as failed.

Addresses that never respond

Sometimes an address has never responded at all. It may contain a typo, or the site may only be reachable from inside a private network. There is no point scanning an unreachable address again and again, so we stop scanning the page and email you. We check the address once a day and turn monitoring back on as soon as it responds.

A page that used to load and then stops responding is handled the other way: it stays monitored. Taking a site offline for a few minutes, by accident or on purpose, should not switch monitoring off.

Page discovery, the step that finds a site's important pages for you, follows the same idea now. It used to suggest likely checkout and login addresses even for sites that had never responded. Now it only suggests them for sites that have answered.

How often pages are scanned

Every monitored page is scanned at least once a day. Pages we identify as payment, checkout, cart, login, registration, account, password reset or admin pages are scanned every four hours. Our page discovery finds these pages automatically and recognizes their names in many languages, which matters for shops outside English-speaking markets. We wrote about that in Why Automated Security Monitoring Fails.

What we still can't see

Some bot protection blocks automated visitors however they behave. When we recognize the block page, the result says the page was blocked, and the setup guide names the protection when it can identify it. The fix is to let our scanner through; the guide shows what to allow and checks that the rule works. The rest of this summer's changes are in What Changed in ScriptPatrol During Four Months of Open Beta.

How to check what your scanner actually saw

Whatever tool you use, ask what a blocked scan looks like in its results. If a blocked scan and a clean scan look the same, a green check tells you nothing about the page.

A developer can also test it in a few minutes. Open your checkout in a normal browser, look at the scripts it loads in the developer tools and compare that list with the scripts the scan reported. If the scan lists mostly scripts from your bot protection vendor, or far fewer scripts than the browser shows, it probably did not see the real page.