How We Test a Detection Rule Before It Reaches Your Site

Writing a rule that catches a known skimmer is the easy part. Most of the work goes into keeping it quiet on the legitimate code that runs on ordinary checkout pages. A false alarm on a payment page is expensive: the person who receives it learns that our alerts can be wrong, and the next one gets less attention.

Four sets of legitimate code

Every new rule, and every change that makes an existing rule broader, is tested on more than 3,600 legitimate scripts before release. We keep them in four sets, because each kind of code causes different false alarms:

• live scripts from real websites, such as analytics, tag managers, chat widgets and the sites' own code
• open-source libraries, including large minified bundles with embedded encoded data that simple pattern matching tends to find suspicious
• payment providers' SDKs, whose hosted fields and drop-in forms read card fields and send the data to another server, much like a skimmer
• checkout code that merchants write themselves, for example custom form validation

A rule is released only if it raises no alert on any of these scripts. If a candidate rule fires somewhere, we read the code it fired on before deciding what to do with the rule.

The zero that tested nothing

In July our detector had zero false alarms on more than 1,600 live scripts. We then ran it on legitimate payment code, and it flagged four files. One of them was a widely used drop-in checkout form.

Looking into it, we found that the part of one rule meant to tell skimmers apart from payment code had not matched a single one of those scripts. The zero was accurate, but that part of the rule had never been tested.

We fixed the rules and changed how we test. Hits are now counted separately for each part of a rule, a part that never fires is reported as untested, and payment SDKs became a separate test set.

Tests run on the deployed detector

The tests run against the deployed build itself. Every result is stored with the detector version it was measured on, so when the detector changes, earlier results stop counting until the tests have run again.

Tests that fail without the fix

Each detection change comes with a regression test for both sides: the attack the rule must catch and the legitimate code it must leave alone. Before we rely on a new test, we put the original bug back into the code and check that the test fails, then restore the fix and check that it passes. A test that passes on broken code is worse than no test.

Real attacks we don't tune against

A detector can look very good on the samples it was developed with. So we keep a separate collection of real skimmers from campaigns seen in the wild, and we never write a rule to match a particular file from it. It shows us which kinds of attacks get through.

We don't publish a single detection rate based on it. The collection covers only a handful of attack families, and one percentage would look more precise than the data allows.

When a rule misses

No set of rules catches everything, which is why detection runs on top of change monitoring. We fingerprint the scripts on a monitored page, read each new version in full and record each change whether a rule recognizes it or not. On payment and checkout pages, a script that is new to the page still needs a person's approval.

We also test our own safeguards by altering copies of real scripts the system already trusts, the way an attacker would, and checking that the trust is withdrawn. How all this affects the emails site owners get is described in Why a Good Script Monitor Rarely Emails You.