Re: Packet number encryption

[Kazuho Oku <[e-mail]>]

2018-02-06 9:30 GMT+09:00 Jana Iyengar <[e-mail]>:
> I don't think we're converging here, so I'd like to try and encourage us to
> move towards it. I have my opinion, but I care more about forward progress
> at this point.
>
> There are three designs so far:
> 1. Packet numbers as is, with random gaps around migration.
> 2. Packet numbers encrypted, as per Martin's PR.
> 3. Packet numbers encrypted, plus a small modulo sequence number somewhere.

I wonder if we can provide opportunity to observe upstream loss /
reorder rate by just reserving one bit per packet.

The strawman is as follows.

Sender exposes one bit per packet that can be observed by on-path
devices. The bit is flipped for every 256 packets that the sender
sends. Observer records the bit of each packet as they pass by.

If there is no loss or reorder, the pattern an on-path device would
observe is repetition of 256 zeros and 256 ones. In other words, the
wavelength is 512 samples.

When there is a loss, the wavelength becomes shorter. The wavelength
becomes 512 * (1-loss_rate) samples. For example, if the wavelength
being observed is 384, loss rate is 25%.

If there is reorder, the edge (i.e. shift from zereos to ones, or vice
versa) becomes unclear. Duration of the transition will reflect the
extent of reordering.

Theoretically, it might be possible to explain the approach as a way
to let the sender emit a wave, then FFT the wave in on-path devices to
observe loss / reorder. Or it could be explained as just sending one
bit of the sequence number, somewhere in the middle (i.e., not LSB nor
MSB).

The pros of the approach is obvious; we will only spend 1 bit per
packet. Privacy impact is minimum. Implementing it is easy. Another
benefit is that it would be difficult (if not impossible) for on-path
devices to use the bit for purposes other than observation (i.e.,
change the ordering of the packets "for performance").

OTOH I am not sure how hard it would be to build tools for observing
upstream behavior using the proposed approach. Or I could well be
missing some requirements that operators have, as I am an endpoint
developer.

>
> The concrete ones here are (1) and (2).  I'm listing (3) but am not inclined
> to consider it seriously, since it's not concrete enough, (and I don't want
> more bits to be spent in the QUIC packet header if that's where this info
> goes.)
>
> There are opinions about ossification, privacy, and manageability. I'm not
> hearing bloody murder, so for what it's worth, I'd like to stipulate first
> that we can all live with the final design, whichever one it is.  If anyone
> disagrees, please say so -- I think it'd be useful to hear if there are
> folks who think a particular design is fundamentally problematic.
>
> My opinion is still for (2) over (1), but I can live with either.
>
> On Mon, Feb 5, 2018 at 3:32 PM, Jana Iyengar <[e-mail]> wrote:
>>
>> Praveen,
>>
>> That was precisely my point about the ecosystem evolving. Now that we have
>> the ability to classify flows, various things have been built around being
>> able to tell TCP flows. This obviously has benefits as you point out, but
>> the downside is that that I can't get good network utilization with a single
>> flow. I understand the scaling point, but speaking of common cases, any
>> server in the wild is usually serving a large number of connections at
>> moderate speeds, not a single one at 25-35 Gbps... which makes that sort of
>> scaling less exciting than in a microbenchmark.
>>
>> FWIW, the other downside of ECMP is that multipath transport doesn't work
>> -- you need to hash on something else besides the 4-tuple. (We had to work
>> around this in Google's deployment of connection migration.)
>>
>> I'll disagree with your point about reordering being the uncommon case.
>> While that's true today, this is again an expectation that the network works
>> hard to maintain, though the ecosystem and what we expect as "usual" would
>> be quite different had we not built TCP's requirements into the network.
>> There's nothing wrong with having n-modal latencies... we can engineer
>> around that. Any sensible load distribution scheme within the network will
>> give you increased variance but it should limit the worst-case latency.
>> You'd perhaps agree that that's a net win.
>>
>> This was an example of how exposing packet number or not can have long
>> term ecosystem effects. My point was about the fact that exposing can cause
>> ossification.
>>
>> - jana
>>
>> On Mon, Feb 5, 2018 at 9:24 AM, Praveen Balasubramanian
>> <[e-mail]> wrote:
>>>
>>> >> This was definitely true for implementations of TCP, but that is TCP's
>>> >> problem, not the network's
>>>
>>> Flow classification happens through the network and also on the end host
>>> with RSS where a flow is mapped to a core. This allows for building high
>>> performance receive processing that could be lock free for the most part.
>>> The only downside of ECMP on the network and RSS on the CPU is that a single
>>> flow will not take multiple paths or get processed on more than one core.
>>> Windows on server machines today can saturate 25-35 Gbps for a single TCP
>>> connection before being CPU limited. This is a  reasonable trade off because
>>> I don’t know of cases where a single flow is driving more than that amount
>>> of traffic. Assumptions about how flows get classified help make for more
>>> efficient processing. They also lead to consistent latency. You do not want
>>> the traffic to take a bi-modal or N-modal paths (on the network or the host)
>>> to being processed because the fluctuations in latency will hurt the
>>> workload. I am not arguing for TCP not being resilient to reordering but IMO
>>> that should be the uncommon case not the common case. With QUIC if streams
>>> were exposed on the network you could take advantage of stream level ECMP
>>> and RSS to scale better than TCP but we have chosen to keep the 4-tuple (or
>>> the 5-tuple) as the flow classifier on the network which is ok by me since
>>> it leads to TCP parity.
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> From: QUIC [mailto:[e-mail]] On Behalf Of Jana Iyengar
>>> Sent: Sunday, February 4, 2018 8:35 PM
>>> To: Salz, Rich <[e-mail]>
>>> Cc: Lubashev, Igor <[e-mail]>; Roberto Peon <[e-mail]>;
>>> QUIC WG <[e-mail]>
>>> Subject: Re: Packet number encryption
>>>
>>>
>>>
>>> Privacy-protection can't be a user choice, for the reasons others have
>>> noted on this thread.
>>>
>>>
>>> That said, my primary argument is for encryption to avoid ossification.
>>> Not that it matters now, but I'll note that much of GQUIC's original
>>> motivation for encrypting headers was to avoid ossification.
>>>
>>>
>>>
>>> I'll reiterate that fields we expose will get ossified and there are
>>> long-term ecosystem effects to this. Let me illustrate this with precisely
>>> the packet number field. Middleboxes commonly assume that a TCP flow can
>>> only handle packets in-order. This assumption comes from the fact that TCP
>>> implementations get poor performance when packets are reordered. This was
>>> definitely true for implementations of TCP, but that is TCP's problem, not
>>> the network's. However, almost all load-balancers I know of now will pin all
>>> packets within a TCP flow to one path, leading to sub-optimal performance in
>>> the network, and destroying incentives for the endpoints to deploy
>>> reordering-resilient TCP implementations (even though there are plenty of
>>> ways of doing this.)
>>>
>>>
>>>
>>> Exposing QUIC's packet number field (as any field) is likely to have
>>> similar consequences and a similar ecosystem arc.
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Feb 4, 2018 at 7:51 PM, Salz, Rich <[e-mail]> wrote:
>>>
>>> > Optional security tends to devolve to non-secure.
>>>
>>>
>>>
>>> That’s a great aphorism.  And sadly all too true.
>>>
>>>
>>
>>
>



-- 
Kazuho Oku