How We Test a Detection Rule Before It Reaches Your Site

When a security tool flags a script on your checkout, you need to be able to trust the alert. Writing a rule that recognizes a known Magecart skimmer (code that copies card details out of a checkout form) is the easy part. Most of the work goes into keeping the rule quiet on the legitimate code that runs on ordinary checkout pages. A false positive, an alert on harmless code, is expensive on a payment page: the person who receives it learns that the alerts can be wrong, and the next one gets less attention.

Four sets of legitimate code

Every new detection rule for malicious JavaScript, and every change that makes an existing rule broader, is tested before release on more than 3,600 legitimate scripts (the count on August 30, 2026). We keep them in four sets, because each kind of code causes different false alarms:

• live scripts from real websites, such as analytics, tag managers, chat widgets and the sites' own code
• open-source libraries, including large files with spaces and line breaks stripped out to save size (minified), which carry blocks of encoded data that simple pattern matching tends to find suspicious
• payment providers' SDKs, the code a payment provider gives shops for its card form, which reads card fields and sends the data to another server, much like a skimmer
• checkout code that merchants write themselves, for example custom form validation

A rule is released only if it raises no alert on any of these scripts. If a candidate rule fires somewhere, we read the code it fired on before deciding what to do with the rule.

The zero that tested nothing

In July our skimmer detection had zero false positives on more than 1,600 live scripts. We then ran it on legitimate payment code, and it flagged four files. One of them was a widely used drop-in checkout form.

Looking into it, we found that the part of one rule meant to tell skimmers apart from payment code had not matched a single one of those scripts. The zero was accurate, but that part of the rule had never been tested.

We fixed the rules and changed how we test. Each test run now reports how many scripts actually reached the part of a rule that tells skimmers apart from legitimate code, and a run where that part was never reached cannot count as clean. Payment SDKs also became a test set of their own.

Tests run on the deployed detector

The tests run against the deployed build itself. Every result is stored with the detector version it was measured on, so when the detector changes, earlier results stop counting until the tests have run again.

Tests that fail without the fix

Each detection change comes with a regression test, an automated test that fails if the problem comes back. It covers both sides: the attack the rule must catch and the legitimate code it must leave alone. Before we rely on a new test, we put the original bug back into the code and check that the test fails, then restore the fix and check that it passes. Without that step a test could also pass on the broken code, and it would report success for a change it does not protect.

Real attacks we don't tune against

A detector can look very good on the samples it was developed with. So we keep a separate collection of real skimmers from Magecart campaigns seen in the wild, and we never write a rule to match a particular file from it. It shows us which kinds of attacks get through.

We don't publish a single detection rate based on it. The collection covers only a handful of attack families, and one percentage would look more precise than the data allows.

When a rule misses

No set of rules catches everything, which is why detection runs on top of change monitoring. We fingerprint every script on a monitored page, read each new version in full whenever we can, and treat a version we could not read in full as unchecked. Each change is recorded whether a rule recognizes it or not.

Every day we also test the rule that decides whether trust in a script still holds. We give it deliberately altered versions of scripts the system already trusts, and check that it refuses each one. How all this affects the emails site owners get is described in Why a Good Script Monitor Rarely Emails You.

What to ask about a vendor's detection

If you are comparing script monitoring tools, two questions tell you a lot about their detection claims.

First, which legitimate code were the rules tested on, and were payment providers' SDKs part of it? A false positive rate measured without the code that handles cards says little about your checkout page.

Second, does the vendor test on real attacks that no rule was tuned against, and how many attack families does that test cover? A detection rate measured on the samples the rules were written for usually looks better than the tool performs against attacks it has never seen.