Re: [tsvwg] Network feedback and security

[Gorry Fairhurst <[e-mail]>]

On 20/10/2023 19:16, Tom Herbert wrote:
> On Fri, Oct 20, 2023 at 8:44 AM Gorry Fairhurst <[e-mail]> wrote:
>> See later,
>>
>> On 20/10/2023 16:12, Tom Herbert wrote:
>>> On Fri, Oct 20, 2023 at 1:11 AM Sebastian Moeller <[e-mail]> wrote:
>>>> Hi Tom,
>>>>
>>>>
>>>>> On Oct 20, 2023, at 01:13, Tom Herbert <[e-mail]> wrote:
>>>>>
>>>>> On Thu, Oct 19, 2023, 7:56 AM Sebastian Moeller <[e-mail]> wrote:
>>>>>> Hi Tom,
>>>>>>
>>>>>>
>>>>>>> On Oct 19, 2023, at 16:41, Tom Herbert <[e-mail]> wrote:
>>>>>>>
>>>>>>> On Thu, Oct 19, 2023 at 12:28 AM Sebastian Moeller <[e-mail]> wrote:
>>>>>>>> Hi Ingemar,
>>>>>>>>
>>>>>>>> On 19 October 2023 07:19:40 CEST, Ingemar Johansson S <[e-mail]> wrote:
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> I have followed this discussi a bit. I focus only on the congestion indication and the discussion around the multi-bit congestion signaling.
>>>>>>>>>
>>>>>>>>> L4S is just recently standardized and we have just starated to get the community onboard to implement and evaluate it on a larger scale.
>>>>>>>>> We should give it some time and get more data before we jump on the next big thing.
>>>>>>>>>
>>>>>>>>> It is of course possible for IETF to start working on the next big thing with multi-bit congestion indication. But as I see it, we currently have more than enough tasks on the table with L4S.
>>>>>>>> [SM] Respectfully, that task is clearly on the proponents of L4S and IMHO should have been done before granting RFC status. L4S already hindered one higher resolution ECN approach (SCE) without actual evidence of superiority, let's leave it at that one victim please.
>>>>>>>> In my opinion L4S is too little too late, as we already know multi-bit per packet information is superior, so we arguably should focus on that.
>>>>>>> Sebastian,
>>>>>>>
>>>>>>> There's a fundamental problem with mulit-bit per packet information:
>>>>>>> there's no place in packets to put the data that will work across the
>>>>>>> Internet! (other than the eight bit TOS/Priority field).
>>>>>>          [SM2] Yes, I agree. The most fitting place would be in the IP header, but there we are out of (re-)usable bits...
>>>>>>
>>>>>>
>>>>>>> The correct
>>>>>>> mechanism for network to hosts signaling is Hop-by-Hop options but
>>>>>>> those currently see a 99% drop rate.
>>>>>>          [SM] Again I agree, for IPv6 any of the extension headers would seems the next best place. It should be one that exists along the full path, and is visible by the end points, if Hop-byHop fits that bill, great. Or not so great, 1% passage is sobering even for a glass half full kind of person.
>>>>>>
>>>>>>
>>>>>>> There are proposed alternatives
>>>>>>> like putting information in transport payload, TCP and now UDP
>>>>>>> options, overloading flow label or IPv6 addresses; but all of these
>>>>>>> have drawbacks that will preclude widescale deployment (these are
>>>>>>> discussed in section 5 of draft-herbert-host2netsig).
>>>>>>          [SM] Yes deployment of any of these including IP6 extension headers is going to be an uphill battle, as the network will only gain little by this, so deployment likely will start in environments where "little" already is worth it, and that I agree is likely not the wider internet. But darn it, it would be great if having to access a service over a congested link that would result in lowish throughput but with acceptable latency, as compared t today's low throughput and high latency and jitter. That might be a way of pitching this, if this allows more graceful operation at/close to saturation this might be interesting for infrastructure folks ()
>>>>>>
>>>>>>> Since network to
>>>>>>> host signaling modifies data in flight limits the solution space even
>>>>>>> more, for instance it might be okay for a switch to read a transport
>>>>>>> payload like QUIC spin bit, but writing data in the transport payload
>>>>>>> risks systematic data corruption.
>>>>>>          [SM] Well, all handling of packets involved reading and writing memory already, and also changes in header fields is performed routinely, TTL/Hop limit come to mind, as does NATP... so there is ample precedence even if we exclude ECN, no?
>>>>>>
>>>>>>
>>>>>>> I agree that multi-bit data should facilitate more precise congestion
>>>>>>> control with faster reaction time, but I also think we need that more
>>>>>>> in the data center than over the Internet and the problem of defining
>>>>>>> a multi-bit packet field for the data is solvable in a limited domain.
>>>>>>          [SM] I respectfully disagree, I do not run a data center, but I do run a home network and use the internet, I think this is a case where all domains would profit. Realistically though I expect this to come to fruition in limited domains/data centers first. Would be sad if it would stay there however.
>>>>> Sebastian,
>>>>>
>>>>> There may be another option. Fortunately, the definition of a limited
>>>>> domain is rather loose. For instance, a provider network could deploy
>>>>> switches that support multi-bit congestion signals in HBH options
>>>>> could be considered a limited domain. So if a host connects to a
>>>>> server in the provider's domain, like a hosted YT server for instance,
>>>>> then the signal is viable end-to-end. The provider network could be
>>>>> quite large and it could partner with connected networks to extend the
>>>>> limited domain. Conceivably, over time the "limited domain" could
>>>>> encompass a significant portion of the Internet. This is part of the
>>>>> plan to extend the reach FAST.
>>>>           [SM2] OK, "starting from a limited domain" seems viable then.
>>>>
>>>>
>>>>> There's a problem with this though: the signal is only viable in the
>>>>> limited domain. For instance, if a packet with HBH options leaves the
>>>>> limited domain then there's a high probability the packet will be
>>>>> dropped. There's no easy means for a host to determine if it's sending
>>>>> within a limited domain, so the sending hosts can't decide on using
>>>>> EH.
>>>>           [SM2] I respectfully disagree, the signal clearly should be end2end and be part of the negotiation then, so endpoints check whether the signal passes between the endpoints and disale its usage if they do not.
>>> Sebastian,
>>>
>>> That was my first intuition as well-- to have hosts do some sort of
>>> "Happy Eyeballs" like probing of path capabilities, and maybe even
>>> creating a mash-up map of the capabilities of paths across the
>>> Internet so that hosts can get a priori information on the path
>>> capabilities to destinations. But thinking it through, I've come to
>>> believe that this is an inordinate amount of complexity and logic we'd
>>> have to add to host stacks just to use a protocol that is defined in
>>> an Internet standard.
>>>
>>> Note that this may coincide with negotiation, but the process itself
>>> is probing the capabilities of the path to each destination.
>>> Presumably, the host could start by sending initial packets with EH
>>> and wait for a response.If packets come back then it knows the path is
>>> viable, if it doesn't get a reply then it knows is that a packet was
>>> dropped somewhere but not why it was dropped (RFC8883 which would
>>> allows routers to send an ICMP error when packets with EH are dropped,
>>> but we know ICMP is a lost cause on the Internet). The host could try
>>> to back off and not use EH to see what happens. But even if the first
>>> packet succeeded, there's no guarantee that all packets will take the
>>> same path and some later packets in the connection may be dropped so
>>> probing might need to be a continuous process over the life of a
>>> connection.
>>>
>>> But there's a more insidious problem to deal with. Instead of dropping
>>> packets, some routers will relegate EH to slow path processing in a
>>> CPU. To the end hosts EH will appear viable, however when that happens
>>> packets will go through different queues and be subject to greater
>>> latency. When packets go through the slow path the likelihood of
>>> congestion increases substantially. So, ironically, by trying to
>>> measure congestion we may be the cause of congestion!
>>>
>>>>> The solution we're proposing is to remove the bits when packets
>>>>> leave the limited domain at an egress router.
>>>>           [SM2] That I disagree with, this needs to be end2end, unless your proposal entails the end points being informed about that removal.
>>> I believe you also mentioned that it is also best effort. If we remove
>>> EH at egress routers then we don't get feedback on congestion for
>>> those destinations, but at least communications are still viable. The
>>> firewalls where this happens already exist and the implementation of
>>> EH removal is very straightforward. IMO, this EH removal is a more
>>> feasible solution than pushing complexity with the necessary probing
>>> logic into all host stacks.
>>>
>>> Tom
>> Somehow this thread seems to have now developed a walk through various
>> topics.
>>
>> Musing on the future direction for IPv6 EH I'is more appropriate to
>> discuss in v6ops (if it's just operationla practice) or 6man (if the
>> desire is really to change the specification of an EH). I'm sure it will
>> get lots more comment there. That sort of discussion isn't really so
>> appropriate here, where the focus ought to be on the end-to-end
>> transport or end-to-network use of these signals.
> Hi Gorry,
>
> Sebastian's proposal is for network devices to produce multi-bit
> signals meant for consumption by the transport layer at end points.
> It's an interesting idea, but it seems reasonable to ask if it's
> feasible for the network layer to provide those signals (right now on
> the Internet it's not). If it's not feasible, then we don't need to
> spend the WG's time considering the transport layer details of the
> solution.
>
> More generally, this isn't only an example of network devices
> generating signals that would be consumed by the transport layer. For
> example, there's also draft-ravi-ippm-csig which was posted to tsvwg
> (in that case the carrier is Ethernet which isn't in scope of IETF).
> We've also seen a couple of drafts proposing to use UDP options to
> convey network layer information in the UDP surplus space. Network to
> host signaling and host to networking signaling inevitably need to
> cross protocol layers, and it's going to be hard if the carrier and
> the content of signals are developed in isolation. If this isn't in
> the charter of tsvwg that's fine, but I don't know that this fits into
> any other WG. Please advise.
>
> Thanks,
> Tom

I see Tom, you're right to ask: where would standards work happen?

A method to transport the signals could be a 6man/intarea topic topic or 
could be a transport one, depending on the design of signal. Some of the 
thoughts seem to go beyond the current definitions and any adaption of 
EH or transport protocols would need to be in the respective groups. I 
think the signals concerning CC or any capacity/latency indications 
would fall in transport. I sympathise with the pain of coordinating 
between WGs - the chairs can help to avoid going backwards and forwards. 
But, the first step is to show the research results, and gather interest 
in deploying a new standard.

TSVWG is now starting to complete work on a 1-but signal in the form of 
L4S.  I see the use of multi-bit CC signals as tempting, and research on 
this topic appears from time-to-time, especially in controlled 
enviroments. That likely would be interesting to discuss in ICCRG, and 
to present prototype experiments. If there's finally a need for a 
standard, then TSVWG would be one place where a spec. could be developed.

Best wishes,

Gorry




Gorry

>
>> Gorry
>>
>> (YSVWG Chgair)
>>
>>>>> as described in
>>>>> draft-herbert-eh-inflight-removal. This also has some nice security
>>>>> properties and coincides with existing network firewalls.
>>>>           [SM2] Will need to read that, when I find time.
>>>>
>>>> Regards
>>>>           Sebastian
>>>>
>>>>
>>>>> Tom
>>>>>
>>>>>>
>>>>>>> Tom
>>>>>>>
>>>>>>>> Regards
>>>>>>>>          Sebastian
>>>>>>>>
>>>>>>>>
>>>>>>>>> /Ingemar
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: tsvwg <[e-mail]> On Behalf Of Christian Huitema
>>>>>>>>>> Sent: Tuesday, 17 October 2023 17:28
>>>>>>>>>> To: Sebastian Moeller <[e-mail]>; [e-mail]
>>>>>>>>>> Cc: [e-mail]
>>>>>>>>>> Subject: [tsvwg] Network feedback and security (was: New Version
>>>>>>>>>> Notification for draft-huang-tsvwg-transport-challenges-00.txt)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 10/17/2023 3:17 AM, Sebastian Moeller wrote:
>>>>>>>>>>> Personally, ever since I read Arslan, Serhat, and Nick McKeown.
>>>>>>>>>> ‘Switches Know the Exact Amount of Congestion’. In Proceedings of the
>>>>>>>>>> 2019 Workshop on Buffer Sizing, 1–6, 2019. I came to the prediction that
>>>>>>>>>> something like max(bufferoccupancy) over a network path in either
>>>>>>>>>> predicted sojourntime or percentual buffer filling could really improve
>>>>>>>>>> congestion control if included in all packets. That paper and follow ups
>>>>>>>>>> on that idea are to my quite convincing. I see no real security concern
>>>>>>>>>> as the network node adding this information might as well have dropped
>>>>>>>>>> the packet if it wanted to harm that flow, so little increase in attack
>>>>>>>>>> surface in that direction. What I hope something like this might allow
>>>>>>>>>> is a gentler exist from slow start (assuming one can measure and compare
>>>>>>>>>> the dynamics of the congestion indicator with those of the slow-starting
>>>>>>>>>> flow to predict when to exist slow start without first having to dump ~2
>>>>>>>>>> too much data into the network in one RTT). The network could indirectly
>>>>>>>>>> profit if end-points use this information to better reign in congestion
>>>>>>>>>> pro-actively, but that clearly is not guaranteed, especially over the
>>>>>>>>>> open internet.
>>>>>>>>>>
>>>>>>>>>> Yes, network feedback is useful. You mention "max(bufferoccupancy) over
>>>>>>>>>> a network path" as a signal, but I would argue that ECN is pretty much a
>>>>>>>>>> 1 bit version of exactly that. So your argument is really for "more ECN
>>>>>>>>>> bits". That's plausible, but the whole "L4S/Prague" effort seems to
>>>>>>>>>> indicate that 1 bit is enough, because you can observe that bit on many
>>>>>>>>>> consecutive packets. The conservative side of me would say we should
>>>>>>>>>> first deploy the 1 bit version, and then study whether we really need
>>>>>>>>>> many more bits.
>>>>>>>>>>
>>>>>>>>>> The "gentler exit from slow start" issue is largely addressed by
>>>>>>>>>> Hystart, which uses the variations of the RTT as a signal. And yes,
>>>>>>>>>> Hystart can be improved by also monitoring the ECN bits, not just the
>>>>>>>>>> RTT.
>>>>>>>>>>
>>>>>>>>>> But there are in fact security concerns. We already know about ICMP
>>>>>>>>>> attacks in which attackers spoof the network feedback and inject fake
>>>>>>>>>> ICMP packets, for example to disrupt PMTU discovery. "Man on the side"
>>>>>>>>>> attackers can easily send a second copy of a packet with modified header
>>>>>>>>>> bits and race it to the destination. If transports accept that copy as
>>>>>>>>>> genuine, they will heed the faked congestion signals and the attacker
>>>>>>>>>> will succeed in disrupting the connection.
>>>>>>>>>>
>>>>>>>>>> -- Christian Huitema
>>>>>>>> --
>>>>>>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>>>>>>>