Re: [Int-area] Adam Roach's Discuss on draft-ietf-intarea-provisioning-domains-10: (with DISCUSS and COMMENT)

[Tommy Pauly <[e-mail]>]

Hi Adam,

Thanks for the feedback! The updated paragraph in the retrieval section, to indicate a maximum failure count per attachment, is:

If the request for PvD Additional Information fails due to a TLS error,
an HTTP error, or because the retrieved file does not contain valid PvD JSON,
hosts MUST close any connection used to fetch the PvD Additional Information,
and MUST NOT request the information for that PvD ID again for the duration
of the local network attachment. If a host detects 10 or more such failures
to fetch PvD Additional Information, the local network is assumed to be
misconfigured or under attack, and the host MUST NOT make any further
requests for PvD Additional Information, belonging to any PvD ID, for
the duration of the local network attachment. For more discussion, see {{security}}.

I've also expanded the security considerations DoS section as follows:

An attacker generating RAs on a local network can use the H-flag and the PvD ID
to cause hosts on the network to make requests for PvD Additional Information
from servers. This can become a denial-of-service attack, in which an attacker
can amplify its attack by triggering TLS connections to arbitrary servers in response
to sending UDP packets containing RA messages. To mitigate this attack, hosts
MUST:

- limit the rate at which they fetch a particular PvD's Additional Information;
- limit the rate at which they fetch any PvD Additional Information on a given local
network;
- stop making requests for a PvD ID that does not respond with valid JSON;
- stop making requests for all PvD IDs once a certain number of failures is reached
on a particular network.

Details are provided in {{retr}}. This attack can be targeted at generic web servers,
in which case the host behavior of stopping requesting for any server that doesn't
behave like a PvD Additional Information server is critical. Limiting requests for
a specific PvD ID might not be sufficient if the attacker changes the PvD ID values
quickly, so hosts also need to stop requesting if they detect consistent failure when
on a network that is under attack. For cases in which an attacker is pointing hosts at
a valid PvD Additional Information server (but one that is not actually associated
with the local network), the server SHOULD reject any requests that do not originate
from the expected IPv6 prefix as described in {{serverop}}.

For the delay calculation, you make a good point that the larger values get pretty unnecessarily large! I'm a bit concerned about making the minimum fetch range be ~4 seconds, as that could end up being user visible for some valid scenarios. How about making the formula "2**(10 + Delay)":

The target time for the delay is calculated
as a random time between zero and 2**(10 + Delay) milliseconds,
where 'Delay' corresponds to the 4-bit unsigned integer in
the last received PvD Option.

This limits it to 1 second as what the RA can request for fastest frequency bound. This isn't incredibly fast, and with the overall limits for how many requests can be made by a client (which provide the larger portion of the DoS prevention, I'd argue), I think this strikes a good balance between usability and precaution. Thoughts?

I've updated the GitHub text for anyone wanting to see the full flow: https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25

Thanks,
Tommy

> On Jan 22, 2020, at 2:58 PM, Adam Roach <[e-mail]> wrote:
> 
> Thanks for the explanation and the further proposed mitigation.
> 
> Allowing the RA to specify an arbitrarily small "Delay" parameter seems to still allow for a pretty big burst of traffic. If I read the proposed interpretation of the "Delay" bits correctly (2**(Delay * 2)), the current behavior is specified to allow a delay upper bound selected from one of the following (approximate) values:
> 
> 1 ms
> 4 ms
> 16 ms
> 64 ms
> 256 ms
> 1 second
> 4 seconds
> 16 seconds
> 1 minute
> 4 minutes
> 17 minutes
> 70 minutes
> 4 hours, 40 minutes
> 18 hours 38 minutes
> 3 days, 3 hours
> 1 week, 5 days
> 
> That's a pretty breathtaking scope, and it's hard to imagine that the first six or so are strictly needed, while all six are in a range that might overload a DDoS target. The final several seem a bit questionable as well, given normal operational timelines for network attachment. If the formula were revised to, e.g., "2**(Delay + 12)" instead of the current formula, you would have an enforced lower bound of roughly four seconds (which should be enough to blunt most DDoS attacks), and an upper bound of roughly 37 hours (which still seems excessive, although not quite as much as the previous upper bound).
> 
> Assuming the additional mitigation you propose below (10 maximum failures per attachment) as well as some means of achieving a lower-bound for "Delay" on the order of multiple seconds, I think I'm good clearing when a new version comes out.
> 
> Thanks for your work in thinking through practical solutions to this issue.
> 
> /a
> 
> On 1/22/20 16:22, Tommy Pauly wrote:
>> Hi Adam,
>> 
>> Thanks for taking a look! I'd like to avoid adding extra checks, such as looking for particular DNS records, to avoid deployment complexity and more opportunities for incomplete configuration. As such, I'd like to dig into this a bit further.
>> 
>> If the attacker in this case is a rogue actor on a local network sending out RAs on their local link, any given attacking host would be restricted in their scope to the devices they can reach. Of course, there could be coordination across many different local networks simultaneously, but that also requires more work on the side of the attacker.
>> 
>> The reason for the delay was to limit the impact on a relatively low-powered and unsophisticated local HTTPS server for serving PvD information, which may itself be on the router. I imagine that any large web server deployment would not have any issue with the load generated from a particular local network. Specifically, if we are limiting any given host to requesting only a few times within a 10 second window on the network at all, and the number of hosts on the network is bounded, the number of opportunities for the attacker to cause load on the servers is limited.
>> 
>> Another option, to avoid remaining concern about hitting wildcarded hosts, is to simply say that if the host keeps receiving PvD IDs with bogus (failing) servers, it disables all fetching of additional information for the duration of the network attachment. Networks that do implement better control over RAs (RA-guard, etc) presumably won't have this issue, and since the additional info is optional, it shouldn't cause any major connectivity issues.
>> 
>> If we require such a limit (you only get to fail to fetch 10 times total per attachment, say), does that mitigate things?
>> 
>> Thanks,
>> Tommy
>> 
>>> On Jan 22, 2020, at 1:51 PM, Adam Roach <[e-mail] <mailto:[e-mail]>> wrote:
>>> 
>>> Thanks! The new text is good, but I don't think it's sufficient. I have two remaining concerns in particular:
>>> The mitigation for wildcarded web hosts appears inadequate, especially given:
>>> The mechanism clearly anticipates a scale where it can generate *single* short torrential burst sufficient to knock an average server over (hence the random delay mechanism for fetching data over HTTP). Given that fact, simple rate-limiting will never be enough if a single tight burst of traffic can be orchestrated.
>>> The more I think about it, the more I believe the TXT-based opt-in solution I proposed in my earlier email is a reasonable approach to protect general-purpose web servers from PvD-client-based attacks.
>>> 
>>> One further comment inline below.
>>> 
>>> /a
>>> 
>>> On 1/22/20 15:17, Tommy Pauly wrote:
>>>> Hi Adam,
>>>> 
>>>> Thanks again for bringing this up! I've updated our text to include mitigations for this attack. It can be found here (https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25 <https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25>), but here's an overview of the proposed text:
>>>> 
>>>> In Section 4.1, I've added two new paragraphs. The first describes time limits on fetching PvD info:
>>>> 
>>>> In addition to adding a random delay when fetching Additional Information, hosts
>>>> MUST enforce a minimum time between requesting Additional Information
>>>> for a given PvD on the same network. This minimum time is RECOMMENDED
>>>> to be 10 seconds, in order to avoid hosts causing a denial-of-service on the
>>>> PvD server. Hosts also MUST limit the number of requests that are made to
>>>> different PvD Additional Information servers on the same network within a short
>>>> period of time. A RECOMMENDED value is to issue no more than five PvD
>>>> Additional Information requests in total on a given network within 10 seconds.
>>>> For more discussion, see {{security}}.
>>>> 
>>>> The second also makes clear the behavior to take in case of failure, which will be the case for non-PvD web servers:
>>>> 
>>>> If the request for PvD Additional Information fails due to a TLS error,
>>>> an HTTP error, or because the retrieved file does not contain valid PvD JSON,
>>>> hosts MUST close any connection used to fetch the PvD Additional Information,
>>>> and MUST NOT request the information for that PvD ID again for the duration
>>>> of the local network attachment. For more discussion, see {{security}}.
>>>> 
>>>> In addition, I added text to the Security Considerations:
>>>> 
>>>> An attacker generating RAs on a local network can use the H-flag and the PvD ID
>>>> to cause hosts on the network to make requests for PvD Additional Information
>>>> from servers. This can become a denial-of-service attack if not mitigated.
>>> 
>>> This doesn't really convey the amplification involved, which I think is highly relevant.
>>> 
>>> 
>>> 
>>>> To mitigate
>>>> this attack, hosts MUST limit the rate at which they fetch a particular PvD's
>>>> Additional Information, limit the rate at which they fetch any PvD Additional
>>>> Information on a given local network, and stop making requests to any PvD ID
>>>> that does not respond with valid JSON. Details are provided in {{retr}}. This attack
>>>> can be targeted at generic web servers, in which case the host behavior of stopping
>>>> requesting for any server that doesn't behave like a PvD Additional Information server
>>>> is critical. For cases in which an attacker is pointing hosts at a valid PvD Additional
>>>> Information server (but one that is not actually associated with the local network),
>>>> the server SHOULD reject any requests that do not originate from the expected IPv6
>>>> prefix as described in {{serverop}}.
>>>> 
>>>> The existing text referenced here about server behavior is:
>>>> 
>>>> The server providing the JSON files SHOULD also check whether the
>>>> client address is contained by the prefixes listed in the additional
>>>> information, and SHOULD return a 403 response code if there is no
>>>> match.
>>>> 
>>>> Let me know if this addresses your concerns!
>>>> 
>>>> Best,
>>>> Tommy
>>>> 
>>>>> On Jan 21, 2020, at 9:26 PM, Adam Roach via Datatracker <[e-mail] <mailto:[e-mail]>> wrote:
>>>>> 
>>>>> ----------------------------------------------------------------------
>>>>> DISCUSS:
>>>>> ----------------------------------------------------------------------
>>>>> 
>>>>> Thanks to the authors and working group for their work on this document.  I
>>>>> have one major concern about the ability for this mechanism to be abused to
>>>>> form DDoS attacks, described below. Unfortunately, while I have identified the
>>>>> attack, I don't have an easy solution to propose that mitigates it satisfactorily.
>>>>> 
>>>>> I also have a handful of mostly editorial comments on the document.
>>>>> 
>>>>> ---------------------------------------------------------------------------
>>>>> 
>>>>> §6:
>>>>> 
>>>>> I was expecting to see a discussion of the DDoS attack that may result from a
>>>>> large network (or a rogue host on such a network) sending out a PvD ID
>>>>> containing the hostname of a victim machine, and setting the "H" flag.
>>>>> 
>>>>> Since the messages used to trigger these HTTP connections are extremely
>>>>> lightweight, unauthenticated UDP messages, and the resulting HTTP connections
>>>>> require the exchange of a significant number of packets in addition to a
>>>>> number of cryptographic operations, this is a very high ratio amplification
>>>>> attack, both in terms of network and CPU resources.
>>>>> 
>>>>> Given that the delay setting comes from the network instead of being
>>>>> independently computed by the host, such an attack could be honed to be
>>>>> particularly devastating.  Although it isn't a complete mitigation, one
>>>>> approach to consider would be moving computation of the delay upper bound to
>>>>> the host, or specifying a minimum upper bound of several minutes (where a
>>>>> smaller value will cause the host to use this minimum upper bound).
>>>>> 
>>>>> Regardless of how this is ultimately handled, I think this is a pretty severe
>>>>> risk that needs addressing in the document prior to publication.
>>>>> 
>>>> 
>>> 
>> 
>