Re: [Int-area] Adam Roach's Discuss on draft-ietf-intarea-provisioning-domains-10: (with DISCUSS and COMMENT)

Tommy Pauly <tpauly@apple.com> Wed, 22 January 2020 22:22 UTC

Return-Path: <tpauly@apple.com>
X-Original-To: int-area@ietfa.amsl.com
Delivered-To: int-area@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 53018120091; Wed, 22 Jan 2020 14:22:14 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.001
X-Spam-Level:
X-Spam-Status: No, score=-2.001 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DBNVSY6YCFvG; Wed, 22 Jan 2020 14:22:11 -0800 (PST)
Received: from ma1-aaemail-dr-lapp03.apple.com (ma1-aaemail-dr-lapp03.apple.com [17.171.2.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 543941200FD; Wed, 22 Jan 2020 14:22:11 -0800 (PST)
Received: from pps.filterd (ma1-aaemail-dr-lapp03.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp03.apple.com (8.16.0.27/8.16.0.27) with SMTP id 00MMBpMd043827; Wed, 22 Jan 2020 14:22:09 -0800
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=sender : from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=EF+dnOXKO6qkzWpd+/KLlVJXFUURo68tsfbAE7fQc9k=; b=CWSq5B62M9zH6qozYXLYNyx++iaI1HlWu4+TSRaPn+DS33kEOaS/GuVGvOjKu/6pnX8l Fe8DUDyWeZ8YluZd2iD4byOwtj7ecn6qqxGm8RQYvmB55HG50o8drdy3nXH6Tcm7jMCS Ih+nqvx00hgI8lQxjYVW5Ng5WM+ZzzgExkyhV5xCWeI1sKx8UUw4HHQotIPnzKVdOgqK FpYg7dqF96uHk34V8x6s8c0OshYz1MiaQs3sRzRvWMohL4Go5GPFBpGId+CLvGLEV60H LmM6pMYwNFIjanUgRTjaOey3vh4p5rbHBjww0l+UYo+hS1TPYLs4ksR89lU9uFznMIO4 7g==
Received: from ma1-mtap-s01.corp.apple.com (ma1-mtap-s01.corp.apple.com [17.40.76.5]) by ma1-aaemail-dr-lapp03.apple.com with ESMTP id 2xm1w18tn3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Wed, 22 Jan 2020 14:22:09 -0800
Received: from nwk-mmpp-sz10.apple.com (nwk-mmpp-sz10.apple.com [17.128.115.122]) by ma1-mtap-s01.corp.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) with ESMTPS id <0Q4J007P84SW3C10@ma1-mtap-s01.corp.apple.com>; Wed, 22 Jan 2020 14:22:09 -0800 (PST)
Received: from process_milters-daemon.nwk-mmpp-sz10.apple.com by nwk-mmpp-sz10.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) id <0Q4J00G004FALM00@nwk-mmpp-sz10.apple.com>; Wed, 22 Jan 2020 14:22:08 -0800 (PST)
X-Va-A:
X-Va-T-CD: f3fe944e6c679f2c3d972f4bd0691c8b
X-Va-E-CD: d60a7396dd592520d25d54063e76f07d
X-Va-R-CD: bd63ceb45b8fd24fe4c278576743557a
X-Va-CD: 0
X-Va-ID: 78ec3f6b-8c67-4fa8-a49e-62a32d623565
X-V-A:
X-V-T-CD: f3fe944e6c679f2c3d972f4bd0691c8b
X-V-E-CD: d60a7396dd592520d25d54063e76f07d
X-V-R-CD: bd63ceb45b8fd24fe4c278576743557a
X-V-CD: 0
X-V-ID: 2c83f5ed-365f-4e30-82b9-aa44a4f196d7
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2020-01-22_08:,, signatures=0
Received: from [17.230.170.238] by nwk-mmpp-sz10.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) with ESMTPSA id <0Q4J00F3B4SWD510@nwk-mmpp-sz10.apple.com>; Wed, 22 Jan 2020 14:22:08 -0800 (PST)
Sender: tpauly@apple.com
From: Tommy Pauly <tpauly@apple.com>
Message-id: <BF4953C0-2502-4E08-B8B3-B55D04475416@apple.com>
Content-type: multipart/alternative; boundary="Apple-Mail=_6287454E-A9DC-40C8-AF85-94CA2E29D43E"
MIME-version: 1.0 (Mac OS X Mail 13.0 \(3594.4.17\))
Date: Wed, 22 Jan 2020 14:22:05 -0800
In-reply-to: <a1daf959-3331-e86d-2734-1f63a98d7625@nostrum.com>
Cc: The IESG <iesg@ietf.org>, ek@loon.com, draft-ietf-intarea-provisioning-domains@ietf.org, int-area@ietf.org, intarea-chairs@ietf.org
To: Adam Roach <adam@nostrum.com>
References: <157967080772.28909.16443816599872682093.idtracker@ietfa.amsl.com> <6AFB6A09-59BF-411D-816F-914BAAF86A9B@apple.com> <a1daf959-3331-e86d-2734-1f63a98d7625@nostrum.com>
X-Mailer: Apple Mail (2.3594.4.17)
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2020-01-22_08:, , signatures=0
Archived-At: <https://mailarchive.ietf.org/arch/msg/int-area/fve8myQduBjjTjAort6w9QJkU1s>
Subject: Re: [Int-area] Adam Roach's Discuss on draft-ietf-intarea-provisioning-domains-10: (with DISCUSS and COMMENT)
X-BeenThere: int-area@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF Internet Area Mailing List <int-area.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/int-area>, <mailto:int-area-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/int-area/>
List-Post: <mailto:int-area@ietf.org>
List-Help: <mailto:int-area-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/int-area>, <mailto:int-area-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 22 Jan 2020 22:22:14 -0000

Hi Adam,

Thanks for taking a look! I'd like to avoid adding extra checks, such as looking for particular DNS records, to avoid deployment complexity and more opportunities for incomplete configuration. As such, I'd like to dig into this a bit further.

If the attacker in this case is a rogue actor on a local network sending out RAs on their local link, any given attacking host would be restricted in their scope to the devices they can reach. Of course, there could be coordination across many different local networks simultaneously, but that also requires more work on the side of the attacker.

The reason for the delay was to limit the impact on a relatively low-powered and unsophisticated local HTTPS server for serving PvD information, which may itself be on the router. I imagine that any large web server deployment would not have any issue with the load generated from a particular local network. Specifically, if we are limiting any given host to requesting only a few times within a 10 second window on the network at all, and the number of hosts on the network is bounded, the number of opportunities for the attacker to cause load on the servers is limited.

Another option, to avoid remaining concern about hitting wildcarded hosts, is to simply say that if the host keeps receiving PvD IDs with bogus (failing) servers, it disables all fetching of additional information for the duration of the network attachment. Networks that do implement better control over RAs (RA-guard, etc) presumably won't have this issue, and since the additional info is optional, it shouldn't cause any major connectivity issues.

If we require such a limit (you only get to fail to fetch 10 times total per attachment, say), does that mitigate things?

Thanks,
Tommy

> On Jan 22, 2020, at 1:51 PM, Adam Roach <adam@nostrum.com> wrote:
> 
> Thanks! The new text is good, but I don't think it's sufficient. I have two remaining concerns in particular:
> The mitigation for wildcarded web hosts appears inadequate, especially given:
> The mechanism clearly anticipates a scale where it can generate *single* short torrential burst sufficient to knock an average server over (hence the random delay mechanism for fetching data over HTTP). Given that fact, simple rate-limiting will never be enough if a single tight burst of traffic can be orchestrated.
> The more I think about it, the more I believe the TXT-based opt-in solution I proposed in my earlier email is a reasonable approach to protect general-purpose web servers from PvD-client-based attacks.
> 
> One further comment inline below.
> 
> /a
> 
> On 1/22/20 15:17, Tommy Pauly wrote:
>> Hi Adam,
>> 
>> Thanks again for bringing this up! I've updated our text to include mitigations for this attack. It can be found here (https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25 <https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25>), but here's an overview of the proposed text:
>> 
>> In Section 4.1, I've added two new paragraphs. The first describes time limits on fetching PvD info:
>> 
>> In addition to adding a random delay when fetching Additional Information, hosts
>> MUST enforce a minimum time between requesting Additional Information
>> for a given PvD on the same network. This minimum time is RECOMMENDED
>> to be 10 seconds, in order to avoid hosts causing a denial-of-service on the
>> PvD server. Hosts also MUST limit the number of requests that are made to
>> different PvD Additional Information servers on the same network within a short
>> period of time. A RECOMMENDED value is to issue no more than five PvD
>> Additional Information requests in total on a given network within 10 seconds.
>> For more discussion, see {{security}}.
>> 
>> The second also makes clear the behavior to take in case of failure, which will be the case for non-PvD web servers:
>> 
>> If the request for PvD Additional Information fails due to a TLS error,
>> an HTTP error, or because the retrieved file does not contain valid PvD JSON,
>> hosts MUST close any connection used to fetch the PvD Additional Information,
>> and MUST NOT request the information for that PvD ID again for the duration
>> of the local network attachment. For more discussion, see {{security}}.
>> 
>> In addition, I added text to the Security Considerations:
>> 
>> An attacker generating RAs on a local network can use the H-flag and the PvD ID
>> to cause hosts on the network to make requests for PvD Additional Information
>> from servers. This can become a denial-of-service attack if not mitigated.
> 
> This doesn't really convey the amplification involved, which I think is highly relevant.
> 
> 
> 
>> To mitigate
>> this attack, hosts MUST limit the rate at which they fetch a particular PvD's
>> Additional Information, limit the rate at which they fetch any PvD Additional
>> Information on a given local network, and stop making requests to any PvD ID
>> that does not respond with valid JSON. Details are provided in {{retr}}. This attack
>> can be targeted at generic web servers, in which case the host behavior of stopping
>> requesting for any server that doesn't behave like a PvD Additional Information server
>> is critical. For cases in which an attacker is pointing hosts at a valid PvD Additional
>> Information server (but one that is not actually associated with the local network),
>> the server SHOULD reject any requests that do not originate from the expected IPv6
>> prefix as described in {{serverop}}.
>> 
>> The existing text referenced here about server behavior is:
>> 
>> The server providing the JSON files SHOULD also check whether the
>> client address is contained by the prefixes listed in the additional
>> information, and SHOULD return a 403 response code if there is no
>> match.
>> 
>> Let me know if this addresses your concerns!
>> 
>> Best,
>> Tommy
>> 
>>> On Jan 21, 2020, at 9:26 PM, Adam Roach via Datatracker <noreply@ietf.org <mailto:noreply@ietf.org>> wrote:
>>> 
>>> ----------------------------------------------------------------------
>>> DISCUSS:
>>> ----------------------------------------------------------------------
>>> 
>>> Thanks to the authors and working group for their work on this document.  I
>>> have one major concern about the ability for this mechanism to be abused to
>>> form DDoS attacks, described below. Unfortunately, while I have identified the
>>> attack, I don't have an easy solution to propose that mitigates it satisfactorily.
>>> 
>>> I also have a handful of mostly editorial comments on the document.
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>>> §6:
>>> 
>>> I was expecting to see a discussion of the DDoS attack that may result from a
>>> large network (or a rogue host on such a network) sending out a PvD ID
>>> containing the hostname of a victim machine, and setting the "H" flag.
>>> 
>>> Since the messages used to trigger these HTTP connections are extremely
>>> lightweight, unauthenticated UDP messages, and the resulting HTTP connections
>>> require the exchange of a significant number of packets in addition to a
>>> number of cryptographic operations, this is a very high ratio amplification
>>> attack, both in terms of network and CPU resources.
>>> 
>>> Given that the delay setting comes from the network instead of being
>>> independently computed by the host, such an attack could be honed to be
>>> particularly devastating.  Although it isn't a complete mitigation, one
>>> approach to consider would be moving computation of the delay upper bound to
>>> the host, or specifying a minimum upper bound of several minutes (where a
>>> smaller value will cause the host to use this minimum upper bound).
>>> 
>>> Regardless of how this is ultimately handled, I think this is a pretty severe
>>> risk that needs addressing in the document prior to publication.
>>> 
>> 
>