RE: Packet number encryption

[Praveen Balasubramanian <[e-mail]>]

Oops sorry I think I misread that 1 requires random jumps only around migration and not always, which means its decidedly inferior to 2 w.r.t. ossification. Can we include a 1.1 which incorporates random jumps even in steady state? Or have we ruled that option out?

From: QUIC [mailto:[e-mail]] On Behalf Of Praveen Balasubramanian
Sent: Monday, February 5, 2018 7:13 PM
To: Jana Iyengar <[e-mail]>
Cc: Salz, Rich <[e-mail]>; Lubashev, Igor <[e-mail]>; Roberto Peon <[e-mail]>; QUIC WG <[e-mail]>
Subject: RE: Packet number encryption

My opinion is for 1 over 2 but I can live with either. The main concern is cost of another encryption / decryption step, so the benefit of 2 over 1 should be demonstrably better on one of ossification, privacy, or manageability.

From: Jana Iyengar [mailto:[e-mail]]
Sent: Monday, February 5, 2018 4:31 PM
To: Praveen Balasubramanian <[e-mail]<mailto:[e-mail]>>
Cc: Salz, Rich <[e-mail]<mailto:[e-mail]>>; Lubashev, Igor <[e-mail]<mailto:[e-mail]>>; Roberto Peon <[e-mail]<mailto:[e-mail]>>; QUIC WG <[e-mail]<mailto:[e-mail]>>
Subject: Re: Packet number encryption

I don't think we're converging here, so I'd like to try and encourage us to move towards it. I have my opinion, but I care more about forward progress at this point.

There are three designs so far:
1. Packet numbers as is, with random gaps around migration.
2. Packet numbers encrypted, as per Martin's PR.
3. Packet numbers encrypted, plus a small modulo sequence number somewhere.

The concrete ones here are (1) and (2).  I'm listing (3) but am not inclined to consider it seriously, since it's not concrete enough, (and I don't want more bits to be spent in the QUIC packet header if that's where this info goes.)

There are opinions about ossification, privacy, and manageability. I'm not hearing bloody murder, so for what it's worth, I'd like to stipulate first that we can all live with the final design, whichever one it is.  If anyone disagrees, please say so -- I think it'd be useful to hear if there are folks who think a particular design is fundamentally problematic.

My opinion is still for (2) over (1), but I can live with either.

On Mon, Feb 5, 2018 at 3:32 PM, Jana Iyengar <[e-mail]<mailto:[e-mail]>> wrote:
Praveen,

That was precisely my point about the ecosystem evolving. Now that we have the ability to classify flows, various things have been built around being able to tell TCP flows. This obviously has benefits as you point out, but the downside is that that I can't get good network utilization with a single flow. I understand the scaling point, but speaking of common cases, any server in the wild is usually serving a large number of connections at moderate speeds, not a single one at 25-35 Gbps... which makes that sort of scaling less exciting than in a microbenchmark.

FWIW, the other downside of ECMP is that multipath transport doesn't work -- you need to hash on something else besides the 4-tuple. (We had to work around this in Google's deployment of connection migration.)

I'll disagree with your point about reordering being the uncommon case. While that's true today, this is again an expectation that the network works hard to maintain, though the ecosystem and what we expect as "usual" would be quite different had we not built TCP's requirements into the network. There's nothing wrong with having n-modal latencies... we can engineer around that. Any sensible load distribution scheme within the network will give you increased variance but it should limit the worst-case latency. You'd perhaps agree that that's a net win.

This was an example of how exposing packet number or not can have long term ecosystem effects. My point was about the fact that exposing can cause ossification.

- jana

On Mon, Feb 5, 2018 at 9:24 AM, Praveen Balasubramanian <[e-mail]<mailto:[e-mail]>> wrote:
>> This was definitely true for implementations of TCP, but that is TCP's problem, not the network's
Flow classification happens through the network and also on the end host with RSS where a flow is mapped to a core. This allows for building high performance receive processing that could be lock free for the most part. The only downside of ECMP on the network and RSS on the CPU is that a single flow will not take multiple paths or get processed on more than one core. Windows on server machines today can saturate 25-35 Gbps for a single TCP connection before being CPU limited. This is a  reasonable trade off because I don’t know of cases where a single flow is driving more than that amount of traffic. Assumptions about how flows get classified help make for more efficient processing. They also lead to consistent latency. You do not want the traffic to take a bi-modal or N-modal paths (on the network or the host) to being processed because the fluctuations in latency will hurt the workload. I am not arguing for TCP not being resilient to reordering but IMO that should be the uncommon case not the common case. With QUIC if streams were exposed on the network you could take advantage of stream level ECMP and RSS to scale better than TCP but we have chosen to keep the 4-tuple (or the 5-tuple) as the flow classifier on the network which is ok by me since it leads to TCP parity.

Thanks

From: QUIC [mailto:[e-mail]<mailto:[e-mail]>] On Behalf Of Jana Iyengar
Sent: Sunday, February 4, 2018 8:35 PM
To: Salz, Rich <[e-mail]<mailto:[e-mail]>>
Cc: Lubashev, Igor <[e-mail]<mailto:[e-mail]>>; Roberto Peon <[e-mail]<mailto:[e-mail]>>; QUIC WG <[e-mail]<mailto:[e-mail]>>
Subject: Re: Packet number encryption

Privacy-protection can't be a user choice, for the reasons others have noted on this thread.

That said, my primary argument is for encryption to avoid ossification. Not that it matters now, but I'll note that much of GQUIC's original motivation for encrypting headers was to avoid ossification.

I'll reiterate that fields we expose will get ossified and there are long-term ecosystem effects to this. Let me illustrate this with precisely the packet number field. Middleboxes commonly assume that a TCP flow can only handle packets in-order. This assumption comes from the fact that TCP implementations get poor performance when packets are reordered. This was definitely true for implementations of TCP, but that is TCP's problem, not the network's. However, almost all load-balancers I know of now will pin all packets within a TCP flow to one path, leading to sub-optimal performance in the network, and destroying incentives for the endpoints to deploy reordering-resilient TCP implementations (even though there are plenty of ways of doing this.)

Exposing QUIC's packet number field (as any field) is likely to have similar consequences and a similar ecosystem arc.


On Sun, Feb 4, 2018 at 7:51 PM, Salz, Rich <[e-mail]<mailto:[e-mail]>> wrote:

> Optional security tends to devolve to non-secure.



That’s a great aphorism.  And sadly all too true.