Re: [secdir] SECDIR Review draft-koster-rep

[tirumal reddy <[e-mail]>]

On Wed, 22 Jun 2022 at 21:06, Gary Illyes <[e-mail]> wrote:

> Thanks Tiru!
>
> I updated our public repository with your suggestions and a diff of the
> changes can be seen at
> https://github.com/google/robotstxt/commit/a048272f9091570db556cf3656b6d33250797bba
>
> Specifically on point c) we added a new paragraph with a list of vectors
> related to implementors based on a conversation we had with our security
> team. On point a) and b) we restated that robots.txt is not a security
> measure whatsoever and folks should employ a valid security measure such as
> IP based ACL
>

Thanks Gary, changes look good to me. Are there better security measures to
refer to than the IP based ACL (IP reputation is challenging with IPv6
addresses and the IP address can possibly be spoofed) ?

-Tiru


>
> On Mon, Jun 20, 2022 at 1:51 PM tirumal reddy <[e-mail]> wrote:
>
>> SECDIR Review draft-koster-rep
>>
>>
>> Reviewer: Tirumaleswar Reddy
>> Review result: Ready with Issues
>>
>>
>>
>> I have reviewed this document as part of the security directorate's
>>
>> ongoing effort to review all IETF documents being processed by the
>>
>> IESG..  Document editors and WG chairs should treat these comments
>>
>> just like any other last call comments.
>>
>>
>>
>> You may want to discuss the following security threats:
>>
>>
>>
>> a) Revealing disallowed URIs will make its paths easily discoverable.
>> However, security by obscurity will not maintain or increase the security
>> of the content provider (you can refer to
>> https://datatracker.ietf.org/doc/html/rfc4949).
>>
>> b) A malicious crawler will not honor the disallow rules and can try to
>> access the disallowed URIs, it should be mitigated by access control
>> restrictions. Discuss any other count-measures used to block such malicious
>> crawlers (like blocking the IP address).
>>
>> c) Attacks possible on crawlers because of a malicious robots.txt file.
>>
>>
>> Cheers,
>>
>> -Tiru
>>
>