Re: [dispatch] Proposal for scantxt; scanning opt-in/out, identification, verification, notification, and reporting

[John Levine <[e-mail]>]

It appears that Ollie IETF  <[e-mail]> said:
>-=-=-=-=-=-
>Hi all,
>
>The scantxt project (https://www.scantxt.org / https://github.com/scantxt) I'm pulling together aims to develop a
>set of mechanisms for both scan recipients (website/infrastructure operators) to indicate their preferences and
>verify scans, and scanning tools to identify themselves and report findings in a consistent way.

I have to ask who would be interested in something like this.

robots.txt is a special case since most web sites welcome or at least
tolerate being included in web directories, so there is an incentive
to allow spidering, and for spiders to avoid being annoying when they
scan. It's also special because it is generally very easy to add a
robots.txt file to a web server without adding any code or
configuration changes.

Other than that, the vast majority of scanning is malicious. Or even
if it purports to be benign or for research, as often as not the
"researchers" are lying, or they are poorly supervised students trying
to collect data for yet another low quality paper. When I see a scan
the most I am inclined to do is see who owns the network (a university
I recognize is better, a random AWS address is worse), and whether
there is a web server on the address explaining what they're doing and
offering no-hassle no-question network-wide opt-out.

Then I adjust my packet filters to block it.

R's,
John