Re: Is the invariants draft really standards track?

[Martin Duke <[e-mail]>]

Ah, I understand now. Thanks for explaining it again.

Oddly enough, the load-balancing part of QUIC-LB provides some DoS
protection by making it easier to check for valid conn IDs on 1RTT packets
further away from the server.

On Fri, Jun 19, 2020 at 4:52 PM Christian Huitema <[e-mail]>
wrote:

>
> On 6/19/2020 4:00 PM, Martin Duke wrote:
>
> >  The problem of the DOS protection box is to drop as much traffic as
> possible while still letting some good in, and also without causing
> performance hits when there are no attacks. There are indeed some thorny
> traffic management issues there. More information helps, and Ben's
> suggestion to look at DOTS is certainly on point.
>
> I am not sure how any of this can work when inbound 1RTT packets can come
> with essentially random IP/port and CID and potentially be valid, given
> there might have been a migration. Some of the QUIC-LB modes at least make
> it hard to guess your way to a valid CID.
>
> The DOS box does not have to worry about what kind of traffic is coming
> in. It just has to open a context for the 5 tuple, and check whether it
> sees 1-RTT packets coming back. And then maybe count the volume of 1-RTT
> packets coming back.
>
> The worry is that one of the bots might start a legitimate connection,
> then disclose its five tuple to the rest of the botnet. The whole botnet
> can then spoof the 5 tuple that was just pin-holed by the DOS box. A simple
> "open-close" logic is thus not good enough. The DOS box must also enforce
> some kind of rate limiting per 5 tuple.
>
> Which also means that if a botnet can predict the 5 tuple used by a
> legitimate connection and then spoof it, it can DOS it. Once you start
> digging that particular rabbit hole, the joy never stops...
>
> -- Christian Huitema
>
>
>
> DOTS is an intriguing idea, but again I'm not sure what signatures one
> could use to configure it.
>
> On Fri, Jun 19, 2020 at 3:43 PM Christian Huitema <[e-mail]>
> wrote:
>
>>
>> On 6/19/2020 3:05 PM, Ted Hardie wrote:
>>
>> Hi Christian,
>>
>> A question in-line,
>>
>> On Fri, Jun 19, 2020 at 2:49 PM Christian Huitema <[e-mail]>
>> wrote:
>>
>>> When under DOS attack, you want to "minimize blowback", i.e., as much as
>>> possible avoid generating packets in response to attack traffic. So, yes, a
>>> server may choose to not send stateless resets to anyone when under attack;
>>> in fact, my recommendation would be that a server SHOULD NOT send stateless
>>> resets to anyone when under attack.
>>>
>>> That said, Igor raised an interesting point about return traffic. It
>>> would be very nice if DOS protection boxes could distinguish between
>>> "validated traffic" that the server presumably intends to process, and
>>> "unsolicited traffic" that will just consume resource. The box can then
>>> reserve some share of the resource for validated traffic, and place the
>>> rest of the traffic in a lower priority queue. Fine, but there needs to be
>>> a test. The classic test is that incoming traffic is "validated" if the
>>> protection box can match it with return traffic coming from the server --
>>> for some definition of matching.
>>>
>>> From that point of view, stateless reset is definitely not helpful. But
>>> problematic traffic goes beyond that. The server will reply to a client's
>>> initial with a server's initial packet. Does that validate the response
>>> traffic? OK, maybe the protection box can programmed to only validate
>>> traffic if it sees 1RTT packets. But many servers will send 1-RTT packets
>>> as 0.5 RTT. Does that validate the response traffic?
>>>
>>> We might say that traffic is validated when the handshake is confirmed,
>>> but the protection box does not understand the TLS handshake, it just sees
>>> packet types and packet sizes. It cannot distinguish between 0.5RTT data
>>> and 1RTT data, and thus the closest approximation of "validation" would be
>>> seeing more than an initial window worth of traffic coming back from the
>>> server. That does not sound great.
>>>
>>> On the other hand, things get much better if the server under attack can
>>> adopt a defensive posture and help the DOS protection box do its job.
>>> Suppose that a server can detect that it is under attack -- or be
>>> explicitly configured so. The simplest defensive posture would be to (1)
>>> disable stateless reset and (2) not send any 0.5RTT packet, including in
>>> response to 0-RTT. The protection boxes can at that point take the 1-RTT
>>> packets from the server as indicating validation.
>>>
>> Perhaps I'm misreading you here, but this sounds like you expect the
>> protection boxes to be able to distinguish when servers see themselves as
>> under DOS from when they don't, so that they can tell that the lack of
>> 0.5RTT is an indication of an attack response.  Given distribution patterns
>> of DOS attacks, I'm struggling to see whether that will commonly be the
>> case.
>>
>> Maybe I wasn't too clear. I don't think that DOS protection boxes are
>> trying to take directions from servers. I also don't believe that DOS
>> protection boxes have any practical means of distinguishing between 0.5 RTT
>> and 1 RTT traffic. That might actually be a problem.
>>
>>
>> You could, of course, always put handshake traffic into low-priority
>> queues until you see the 1-RTT packets that validate the server's
>> interest.  That would make 1-RTT traffic effectively a path signal of
>> validation.
>>
>> Yes, that's more or less what I was considering. But of course it only
>> works if the server refrains from sending 1RTT packets until the handshake
>> is confirmed. Otherwise, the protection boxes will have to count the number
>> of 1RTT packets coming back from the server, or maybe the amount of data
>> coming from the server, and only consider the traffic validated if number
>> of packets and amount of data are larger than common values of IW. You
>> might end up with 3 classes instead of 2 -- validated if 1RTT > IW,
>> validation in progress if 1RTT see but < IW, not validated yet if no 1RTT
>> seen.
>>
>> My concern with that is that using those low priority queues during the
>> handshake phase seems likely to result in worse latency and increased risk
>> of packet loss (which can be tricky during that phase).  That seems a heavy
>> price to pay during non-attack times for better protection when under
>> attack.
>>
>> The problem of the DOS protection box is to drop as much traffic as
>> possible while still letting some good in, and also without causing
>> performance hits when there are no attacks. There are indeed some thorny
>> traffic management issues there. More information helps, and Ben's
>> suggestion to look at DOTS is certainly on point.
>>
>> I am just doing a thought experiment so far. One immediate observation is
>> that in the absence of other data, DOS protection designers are likely to
>> look at packet types, and certainly reason about 1RTT packets versus long
>> header packets. This is definitely an ossification risk.
>>
>> The other observation is that we have to be careful about the realism of
>> thought experiments. If I remember correctly, the payload of the Blaster
>> worm was something like:
>>
>>     On each bot
>>
>>         Open 256 threads
>>
>>             In each thread, loop on "GET some large page from the server"
>>
>> Reasoning about validated traffic would not stop that. But reasoning
>> about validated traffic forces the attacker to disclose the "real" IP
>> addresses of the attacking bots, which then enables a second line of
>> defense.
>>
>> -- Christian Huitema
>>
>>
>>
>>
>> Am I misunderstanding how this is distinguished?
>>
>> Clue appreciated,
>>
>> Ted
>>
>>
>>
>>> Maybe we should specify that.
>>>
>>> -- Christian Huitema
>>> On 6/19/2020 1:42 PM, Lubashev, Igor wrote:
>>>
>>> > There is no need for servers to decrypt CIDs in QUIC-LB. Presumably
>>> the server has a lookup table for its CIDs.
>>>
>>>
>>>
>>> Sending a stateless reset in response to a junk packet would cost more
>>> CPU than verifying CID integrity.  But, yes, a server may choose to not
>>> send stateless resets to anyone when under attack.
>>>
>>>
>>>
>>>
>>>
>>> *From:* Martin Duke <[e-mail]> <[e-mail]>
>>> *Sent:* Friday, June 19, 2020 2:44 PM
>>>
>>> >Unfortunately, Retry system protects only server's memory state and
>>> some CPU cycles spent on crypto.  (Servers still need to decrypt CID to
>>> decide it is invalid, and if the attacker is clever enough to establish one
>>> valid connection and use that CID in a flood, the server will also be
>>> decrypting packets.)
>>>
>>>
>>>
>>> There is no need for servers to decrypt CIDs in QUIC-LB. Presumably the
>>> server has a lookup table for its CIDs.
>>>
>>>
>>>
>>> It is true that Retry Services (and indeed, the Retry concept as a
>>> whole) does nothing to protect network capacity.
>>>
>>>
>>>
>>> On Fri, Jun 19, 2020 at 8:08 AM Lubashev, Igor <[e-mail]>
>>> wrote:
>>>
>>> It looks like
>>> https://tools.ietf.org/html/draft-ietf-quic-load-balancers-02#section-5
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dquic-2Dload-2Dbalancers-2D02-23section-2D5&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=eTT-BoZ1fMitywKRVSxpU3js0lhO0qkspYTKljvj-ys&s=1x7AN2a51X_zV5xSyK0uCL8vZ5cugD3n4lbFWZDqVQg&e=>
>>> is an excellent discussion of Retry mechanics.  It definitely deserves a
>>> reference from this manageability draft.
>>>
>>>
>>>
>>> The Retry mechanisms described in LB draft are all cooperating boxes,
>>> and servers must be aware of them.  Unfortunately, Retry system protects
>>> only server's memory state and some CPU cycles spent on crypto.  (Servers
>>> still need to decrypt CID to decide it is invalid, and if the attacker is
>>> clever enough to establish one valid connection and use that CID in a
>>> flood, the server will also be decrypting packets.)  Retry does nothing to
>>> protect network resources.
>>>
>>>
>>>
>>> The PR I opened (https://github.com/quicwg/ops-drafts/pull/94
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_quicwg_ops-2Ddrafts_pull_94&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=eTT-BoZ1fMitywKRVSxpU3js0lhO0qkspYTKljvj-ys&s=VpvhQ9n79rANIk3O5Sm7PZyToFQEennoPt9iqFpWbq8&e=>)
>>> is about uncooperating devices that try to mitigate volumetric network
>>> attacks.
>>>
>>>
>>>
>>>
>>>
>>> *From:* Martin Duke <[e-mail]>
>>> *Sent:* Wednesday, June 17, 2020 8:16 PM
>>>
>>> Hi Igor, you might want to check out the "Retry Services" bit of the
>>> QUIC-LB draft. This has something to do with the DDoS use case you discuss.
>>>
>>>
>>>
>>> On Wed, May 27, 2020 at 9:07 AM Lubashev, Igor <[e-mail]>
>>> wrote:
>>>
>>> I’m working on a manageability draft PR for this (how to rate limit UDP
>>> to reduce disruption to QUIC if you have to rate limit UDP).  ETA end of
>>> the week (if I do not get pulled into something again).
>>>
>>>
>>>
>>> The relevant observation is that DDoS with UDP that is indistinguishable
>>> from QUIC will happen.  UDP is already the most prevalent DDoS vector,
>>> since it is easy for a compromised non-admin app to send a flood of huge
>>> UDP packets (with TCP you get throttled by the congestion controller).  So
>>> there WILL be DDoS protection devices out there to try to mitigate the
>>> problem, possibly by observing both directions of the flow and deciding
>>> whether a packet belongs to a valid flow or not.
>>>
>>>
>>>
>>> Since such middle boxes will be created, the more explicit and normative
>>> Invariants are about what one can expect, the less such middle boxes may
>>> decide for themselves.  For example (I did not think long about it), if
>>> some elements of path validation could land into Invariants (roughly, “no
>>> more than X packets/bytes can be sent on a new path w/o a return packet”),
>>> a DDoS middle box may use this fact and active connection migration might
>>> still have a chance during an attack (NAT rebinding could be linked by DDoS
>>> boxes to an old connection via unchanged CID).
>>>
>>>
>>>
>>>    - Igor
>>>
>>>
>>>
>>>
>>>
>>> *From:* Christian Huitema <[e-mail]>
>>> *Sent:* Wednesday, May 27, 2020 11:34 AM
>>>
>>> On 5/27/2020 8:28 AM, Kyle Rose wrote:
>>>
>>> On Wed, May 27, 2020 at 10:34 AM Ian Swett <ianswett=
>>> [e-mail]> wrote:
>>>
>>> I was agreeing with MT, but I'm happy to see some more MUSTs added if
>>> people feel that'd be helpful.
>>>
>>>
>>>
>>> Coincidentally, we were just talking about this internally at Akamai
>>> yesterday. IMO, an invariants document isn't really helpful if it isn't
>>> normative, and for it to be normative it (or a related practices doc for
>>> operators) really needs to spell out clear boundaries for operators with
>>> MUSTs..
>>>
>>>
>>>
>>> The example that came up yesterday was around operators filtering QUIC
>>> in the event of a DDoS: one recommendation based on some conversations
>>> going back at least to Prague 2019 was to hash packets on 4-tuple and
>>> filter those below a hash value chosen for a desired ingress limit instead
>>> of doing what most operators do with UDP today, which is to cap UDP
>>> throughput and just drop packets randomly or tail drop.
>>>
>>> Interesting. Did they consider using the CID, or a fraction of it? This
>>> looks entirely like the scenario for which we developed stateless retry.
>>>
>>>
>>>
>>> This recommendation certainly imposes some constraints on future
>>> protocol development that motivate new invariants: for instance, it would
>>> preclude sharding a connection across multiple source ports (not that there
>>> is necessarily a reason to do this; it's just an example). But more
>>> importantly, it goes beyond invariants: it's one among many practices
>>> compatible with the current set of invariants, some reasonable and some
>>> terrible.
>>>
>>> This would break the "preferred address" redirection. Preferred address
>>> migration may or may not be spelled out in the invariants.
>>>
>>>
>>>
>>> Operators are going to do things to QUIC traffic, so it would be good to
>>> offer them recommendations that are compatible with broad deployability.
>>>
>>>
>>>
>>> Yes, we do need the invariants for that.
>>>
>>> -- Christian Huitema
>>>
>>>