Crawler ScriptPatrol Indexu

The ScriptPatrol Index crawler

This crawler is run by ScriptPatrol. For the ScriptPatrol Index it measures the public pages of Czech and Slovak online shops once a quarter: which third-party code they load and how they handle cookie consent. It also looks at the basic security of the website and the domain. Results come out only as aggregate figures, and no shop is named in them.

If you saw it in your server’s log, this page explains what it asks for and how to switch it off for your domain.

What it asks for

First it looks for shops. It sends one request to the homepage of every website on its list and follows the redirects the answer points to. It keeps at most 2 MB of the answer. We build the list of Czech and Slovak websites from public sources. Most websites get nothing more from us in a quarter.

Websites that look like a shop from that first answer, and shops we already know, are then visited by a browser the way a new customer would visit them. It opens the homepage, the login page and the cart, and sometimes the contact page and the terms. It adds nothing to the cart, buys nothing, does not log in and submits no form. We visit a small random sample of the other websites the same way.

Shops then get a second visit. We try the cookie consent banner and the forms the way a customer would. We submit nothing and log in nowhere.

For shops and websites that look like one, we also look at the basic security of the domain and the server as anyone on the internet sees it: DNS records, the homepage’s headers, the security.txt file and the TLS connection.

When a visit fails, for example because the server is overloaded, we try again later, once at most. If we cannot read the homepage, for example because the server refused the visit, we try it with other browsers, one after another, so it may load a few more times.

We visit some shops once more to check that our measurements are accurate.

The first step sends requests to one server address one at a time, at least 1.5 seconds apart. When a server tells more than one website that we ask too often (status 429), we leave it alone for at least an hour and ask the rest of its websites later. We visit shops in random order, so the visits to shops on one hosting provider do not all arrive at once.

In the stored scripts we also look for signs of an attack, such as code that sends details from payment or login forms. When we confirm an attack, we tell the shop’s operator and the national CERT of its country.

When

Once a quarter. Visits to shops start on February 16, May 16, August 16 and November 16 and take several days. Finding new shops, that is the first step and the browser visits to websites that look like shops from it, can run in the weeks before.

Besides that, we visit small samples of websites when we test the Index or change its code. So some websites get a request or a visit from us more than once a quarter.

How to recognize it

Over IPv6, the first step identifies itself with this user agent:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36 (compatible; ScriptPatrolIndex/1.0; +https://scriptpatrol.com/index-crawler)

You can find the first step’s requests over IPv6 by the ScriptPatrolIndex token, without matching the whole string. Over IPv4 the first step identifies itself as an ordinary browser, without the token. The other steps do not carry the token at all. They identify themselves as an ordinary browser, so the shop shows them the same page it shows a customer. Below is what you will see from each step.

When we reach you over IPv6, it is always from 2a01:4f8:1c1a:6d2c::100. Its reverse DNS record points to index-crawler.scriptpatrol.com, and that name points back to the same address. You can check it like this:

$ dig -x 2a01:4f8:1c1a:6d2c::100 +short
index-crawler.scriptpatrol.com.
$ dig AAAA index-crawler.scriptpatrol.com +short
2a01:4f8:1c1a:6d2c::100

The first step asks over IPv6 wherever the website supports it, which is about four in ten Czech and Slovak domains. Otherwise, and often in the other steps, we come over IPv4, from an address we do not publish.

How to opt out

You opt out by email. Write to [email protected] and name the domain. A whole list is fine too. We add the domain to our opt-out list and confirm it by email. Every step of the Index reads the list before it starts, and the first step checks it again on every redirect. From the next step on, we do not contact your domain ourselves, not even with the first step, and it will not be in later editions of the Index. A step that is already running may still send your domain a request before it ends. The opt-out covers subdomains too, such as www.

An opt-out does not stop requests that another website causes. When a website we visit redirects to your domain or loads files from it, such as a script, an image or a font, our browser asks for them as the browser of any other visitor to that website would.

If you write from an address on another domain, we may ask you to confirm that you manage the domain, for example by an email from an address on it.

You can block the crawler by its IPv6 address or by the ScriptPatrolIndex token. That stops only part of the requests, because only the first step carries the token, and only over IPv6, and over IPv4 we come from an address we do not publish. Writing to us is more reliable.

What we keep

From each step we keep at most the following:

We do not keep cookie values from the browser. A page sometimes sends them on by itself, in the address or the content of a request, and then they are in the request we kept. They are values the page gave our browser, and they concern no visitor. From what we keep we derive facts about each website, such as which third-party services it loads, and from those the aggregate figures.

How long we keep it

36 months lets us compare three years of editions and run a corrected rule over older data too, so that a correction does not break the series. Figures already published name no shop and stay published.

Where the data is and who gets it

The data is on our own server, which we rent from Hetzner Online GmbH in Germany. To run ScriptPatrol and work with the data we also use the services of other IT providers, in the EU and outside it, for example for email and for development and data analysis. We do not publish the data or sell it. Otherwise we pass it on only when we confirm an attack: the evidence then goes to the shop’s operator, the national CERT of its country and, where needed, the platform the attack came through.

Personal data

The Index measures shops. Their pages, DNS records and security.txt files can still contain personal data. Mostly that is the name, company number, address, email and phone number of an operator who is a natural person. It is also the names, photos and words of people who appear on the pages, for example in reviews or among contacts, and whatever is on the homepage of a website that is not a shop. We keep it as the website published it, as part of a page, a record or a file. So it always comes from publicly accessible sources.

The controller is Martin Stach, Prague, Czech Republic, who runs ScriptPatrol ([email protected]). We have not appointed a data protection officer.

We process personal data on the basis of legitimate interest (GDPR Article 6(1)(f)), for two purposes.

The first is the measurement for the ScriptPatrol Index. ScriptPatrol makes its living watching the security of websites, and with the Index it shows how Czech and Slovak shops are doing: which third-party code they load, how they handle cookie consent and how well their website and domain are secured. The results are aggregate figures that name no shop, and we make no decision about anyone based on them. So we protect the data as GDPR Article 89(1) asks for statistical purposes: we do not publish it, and we keep it only as long as stated above.

The second is warning about an attack. When we find code that steals the details of a shop’s visitors, it is in the interest of the shop’s customers and its operator to stop the attack as soon as possible (GDPR Recital 49). An operator we warn is told in that warning how we handle their data and what rights they have.

If you ask, we will send you how we weighed these interests against yours.

Telling each person individually whose data may be on tens of thousands of shops and hundreds of thousands of other websites would take disproportionate effort. That is why we inform through this page (GDPR Article 14(5)(b)).

We make no decisions based solely on automated processing. A person approves every attack report before it is sent.

You have the right to access your data, to have it corrected or erased, and to restrict its processing. Write to [email protected] and give the domain, or the name or company number as the website shows it, so that we can find the records. We answer within one month, free of charge. We delete the records about your domain without asking why. Figures already published do not change, because they name no shop. We may keep evidence of an attack we reported if we need it to establish, exercise or defend legal claims.

You can lodge a complaint with the Czech Office for Personal Data Protection (uoou.gov.cz), the Office for Personal Data Protection of the Slovak Republic (dataprotection.gov.sk), or the supervisory authority of the EU country where you live or work, or where the infringement took place.

Your right to object

You can object to the processing of your data at any time. Write to [email protected]. You do not have to give a reason. We opt your domain out, delete the records about it and do not contact it ourselves again, as with the opt-out described above.

For hosting providers and abuse teams

If the crawler causes you trouble, such as load or a customer’s complaint, write to [email protected]. Send the domains it concerns, the time with its time zone, and the log line. We opt out those domains in the same way as above and reply with what we did.