Re: [secdir] SECDIR Review draft-koster-rep

[Gary Illyes <[e-mail]>]

Thanks Tiru!

I updated our public repository with your suggestions and a diff of the
changes can be seen at
https://github.com/google/robotstxt/commit/a048272f9091570db556cf3656b6d33250797bba

Specifically on point c) we added a new paragraph with a list of vectors
related to implementors based on a conversation we had with our security
team. On point a) and b) we restated that robots.txt is not a security
measure whatsoever and folks should employ a valid security measure such as
IP based ACL

On Mon, Jun 20, 2022 at 1:51 PM tirumal reddy <[e-mail]> wrote:

> SECDIR Review draft-koster-rep
>
>
> Reviewer: Tirumaleswar Reddy
> Review result: Ready with Issues
>
>
>
> I have reviewed this document as part of the security directorate's
>
> ongoing effort to review all IETF documents being processed by the
>
> IESG..  Document editors and WG chairs should treat these comments
>
> just like any other last call comments.
>
>
>
> You may want to discuss the following security threats:
>
>
>
> a) Revealing disallowed URIs will make its paths easily discoverable.
> However, security by obscurity will not maintain or increase the security
> of the content provider (you can refer to
> https://datatracker.ietf.org/doc/html/rfc4949).
>
> b) A malicious crawler will not honor the disallow rules and can try to
> access the disallowed URIs, it should be mitigated by access control
> restrictions. Discuss any other count-measures used to block such malicious
> crawlers (like blocking the IP address).
>
> c) Attacks possible on crawlers because of a malicious robots.txt file.
>
>
> Cheers,
>
> -Tiru
>