When a Scan Can't See the Page, It Should Say So

A monitoring tool can be wrong in two ways. It can raise an alarm when nothing is wrong, or it can report that a page is fine when it never saw the page. The second mistake is worse, because nobody investigates a page that looks healthy. Here is how we keep from reporting a clean result for a page we could not check.

Challenge pages

Bot protection from Cloudflare, Akamai, F5, Imperva, DataDome and other vendors sometimes answers an automated visitor with a challenge page instead of the website. Some of them return a normal status code, ordinary HTML and scripts of their own. A scanner that only checks whether the request succeeded can end up storing the challenge page as if it were the checkout.

We recognize challenge pages by their content and mark the scan as blocked. We don't compare it with the page's baseline or use it as the baseline, and it can't produce a clean result.

Site owners can allow our scanner through their firewall. If they do, and a challenge page shows up again later, we mark the rule as no longer working and let them know.

When a check fails

Checks sometimes fail. The connection drops, a request times out or a file arrives incomplete. It is tempting to read a failed check as a negative answer, like “no firewall” or “nothing suspicious”. We record it as not measured:

• A changed script that we could not read in full doesn't count as checked. Once a site is past its first week of monitoring, that alone sends the owner an email.
• If the firewall check ends with an error, it does not report that there is no firewall, and the scan switches to the method we use for protected sites.
• A scan that failed is shown as failed.

Addresses that never respond

Sometimes an address has never responded at all. It may contain a typo, point to an internal server or belong to a site that has moved. There is no point scanning it repeatedly, so we stop scanning the page, email the site owner, check the address once a day and turn monitoring back on as soon as it responds.

A page that used to load and then stops responding is handled the other way: it stays monitored. An outage is a time when someone should be watching, and taking a site offline for a few minutes should not be a way to switch off monitoring.

Discovery follows the same idea now. It used to suggest likely checkout and login addresses even for sites that had never responded. Now it only suggests them for sites that have answered.

How often pages are scanned

Every monitored page is scanned at least once a day. Pages we identify as payment, checkout, cart, login, registration, account, password reset or admin pages are scanned every four hours. Discovery finds these pages automatically and recognizes their names in many languages, which matters for shops outside English-speaking markets. We wrote about that in Why Automated Security Monitoring Fails.

What we still can't see

Some bot protection blocks automated visitors however they behave. The result then says the page was blocked, and the setup guide names the protection when it can identify it. The fix is to let our scanner through; the guide shows what to allow and checks that the rule works. The rest of this summer's changes are in What Changed in ScriptPatrol During Four Months of Open Beta.