Re: [tsvwg] signaling packet importance [was Re: New Version Notification for draft-herbert-fast]

[Dan Wing <[e-mail]>]

On Aug 9, 2023, at 4:36 PM, Tom Herbert <[e-mail]> wrote:
> 
> 
> 
> On Wed, Aug 9, 2023, 2:22 PM Dan Wing <[e-mail] <mailto:[e-mail]>> wrote:
>> Yes, there is a lot of interest in marking 'important' packets.  Thanks for your comments on the list and at the meeting and thanks for updating FAST.  Thanks to Mike for being a nudnik and poking all the co-authors earlier this week, too.
> 
> 
> Hi Dan,
> 
> Thanks for the great discussion! Comments in line.
> 
>> 
>> 
>> 
>> Regarding inapplicability of UDP trailers to TCP, TCP has head of line blocking so marking a packet within a TCP packet differently from other packets in that same flow provides no benefit to the flow.  
> 
> 
> Not necessarily. If a host sees a connection is having problems, like seeing retransmission or high variance in RTT, it's their prerogative to try to affect routing to fix the problem. For instance, this is implemented in Linux as a feature where the IPv6 flow label can be changed mid-flow to see if there's a better route. We can do the same thing for signaling in HBH to affect QoS. While we don't want to be sending every packet differently, the ability to change things occasionally for a TCP connection could be very beneficial.

I wasn't aware of that feature.  Neat.

> 
>> A *separate* TCP flow could get marked differently (e.g., a file transfer TCP flow versus an interactive audio/video TCP flow), but that differential marking is only possible with multiple TCP connections -- causing its drawbacks of NAPT port consumption, connection set-up time, and per-connection congestion control.  Those drawbacks were some of the justification for creating QUIC, and resolved by QUIC.
> 
> 
> Right, connection setup/teardown is expensive which is why there's incentive to try to save failing connections as I described above.
> 
>> 
>> The problem user "A" will mark all their packets as important starving user "B" (the "everyone is an ambulance" problem) requires some sort of permission (or quota or "admission") system -- like FAST.  But that is a problem no matter if IPv6 HbH option, UDP option, DSCP, RSVP, etc.
> 
> 
> Yep, that's a fundamental problem in host network signaling! This also where the ticket analogy is helpful, if concert tickets didn't have features to prevent forgery then everyone would be sitting in the front row for the Taylor Swift concert :-)

Metallica, but yes.


>> 
>> Tom Herbert wrote:
>> 
>>> Can you comment on the requirements and pragmatics of network devices
>>> to process UDP Options. As I mentioned, high performance network
>>> devices process protocol _headers_, it's going to be difficult to
>>> teach them to process trailers efficiently. 
>> 
>> The IP header length and UDP header length are examined to determine if there is a UDP trailer.  Examining the trailer can be helped by encoding "backwards" to ease parsing as was done by SRTP EKT, https://datatracker.ietf.org/doc/html/rfc8870#section-4.1.  
> 
> 
> But I believe that's an E2E protocol. Do you have any examples of protocols with trailers that are explicitly intended to be processed by intermediate nodes?

No, I don't.

> Middle boxes, especially hardware implementation, aren't designed to handle trailers. They may have to defer processing of UDP Options to a software, which as we've seen, time and time, again dooms a data path protocol to the bit bucket! Hosts probably can process trailers, but it's still not nearly as efficient as just processing headers; and host devices with programmable datapath, SmartNICs like P4 programmable programambaility, probably won't be able to process trailers without some major hardware changes. UDP Options might be functional on the Internet, but it's yet to be proven they can be implemented to be sufficiently performant.

Agreed that UDP trailers beg for a software implementation and probably in user mode.  QUIC caused lots of stacks to move to user mode already, but the same incentive is not likely to have occurred on middleboxes yet, if ever.


> 
>> 
>> I agree UDP trailers are not headers, and I agree network devices are quite used to dealing with headers.
> 
>> 
>> The differentiated packet handling would occur near the edge of the network close to the subscriber.  In the case of Wi-Fi it occurs on the subscriber's premises; in the case of 5G it is the radio access network.  Those are where queues build up (due to congestion caused by other users) and where radio interference occurs.  Those devices can improve the user experience by dropping less-important packets in favor of more-important packets.  Today, those devices have little-to-no idea which packets are important.  They know TCP packets will eventually get re-transmitted, and they can make guesses that UDP/443 is carrying QUIC and -- today -- has TCP-like behavior and will also re-transmit packets. But as unreliable QUIC [RFC9221] becomes A Thing, the existing treatment of QUIC by those devices will cause harm.  I believe it is use-cases for unreliable QUIC that is what is causing the influx of new IETF proposals.  Of course, (S)RTP has long wanted this sort of behavior, too, long prior to unreliable QUIC -- it's just been less interesting as the likes of Zoom, Cisco WebEx, Teams, and Skype have tried to quietly just get their job done.  But they, too, could benefit from UDP trailers.
> 
> 
> Reliable protocols need QOS just like any other protocols. Just relying on TCP or QUIC retransmissions would set an awfully low bar for QoS!

Agreed.  

That is why my employer moved from TCP to a (proprietary) UDP-based protocol which is just earning customer deployment this calendar year.  I hope to provide additional incentives to move from our proprietary protocol to QUIC.


> Customers want latency/throughput/jitter guarantees. And if someone is seeing a lot of retransmissions, that's more likely to be a bad thing than a good thing and some action is needed. In E2E protocols, only end hosts have the state to know what's happening to connections, but it's really the network where the problems are and where things can be fixed (like trying to influence routing like described above). That's exactly why we need host network signaling-- the host has the E2E state and understanding of application requirements; the network has the mechanisms and services to satisfy those requirements.

+1.


> 
>> 
>>> If we use fragment options
>>> to force the UDP Options into headers then the problem becomes that we
>>> have to change all end hosts to support that. There's also potential
>>> problems mixing transport and network options, and in particular this
>>> could affect security and DoS considerations.
>> 
>> 
>> Changing OS stacks is already necessary for UDP trailers:  the user- and kernel-mode interfaces to UDP don't provide the flexibility to add a UDP trailer (on Windows); instead, raw sockets are necessary.  Once the entire packet is built by hand with raw sockets, fragment options become viable.  That said, I agree we don't want to make changes to host stacks -- and is one thing I don't like about UDP trailers!
> 
> 
> Yes, frankly that has long been a concern of mine wrt UDP Options. We haven't seen a lot of running code or deployment. I believe there is a FreeBSD patch, but I haven't seen patches sent to netdev for Linux. Neither do I know of any active deployment of UDP Options. And this concern is only about the current use case of transport options, using UDP Options for network consumption hasn't even been brought up until recently. This situation can be contrasted to a protocol like Segment Routing that referenced at least ten implementations, had Linux patches accepted upstream, and I believe there was significant deployment when SR was published as RFC-- see https://datatracker.ietf.org/doc/html/draft-matsushima-spring-srv6-deployment-status-15 for a great example of demonstaring the "running code" requirement of IETF; it is quite thorough in describing the implementation, deployment, and interoperability status of segment routing (I really wish there was an equivalent for UDP Options especially if their scope is being expanded to be processed by network devices).

+1.


> 
>> 
>> 
>> On survivability of UDP trailers:  the sender can probe to determine if UDP trailers survive a path, and avoid sending them the UDP trailer causes failure to receive the packet.  This works because the UDP trailer is an optimization to attempt to help the network deliver improved service.  We have do similar probing for IPv6/IPv4 (Happy Eyeballs), TCP/QUIC, and a slew of other protocols as they are pushed out.  Such probing works fine for UDP trailers because they are unlikely to be dropped on the Internet (that is, on transit networks) because security devices and UDP checksum validation does not occur on such transit networks. 
> 
>> Rather, networks close to the subscriber is where security devices and checksum validation might occur and those networks close to the subscriber are the very same networks where packet priority is more interesting because it's the same place queues build up and packet importance would influence dropping less-important packets.  This creates alignment of the network operator wanting the benefit of UDP trailer (to improve user experience) who can also influence their security vendor and their UDP checksum validation device to permit UDP trailers.
> 
> 
> That same logic can be applied to HBH Options. The ability of operators to offer differentiated services and monetize them, should be sufficient incentive (alignment ) to fix their bugs and make host to network signaling work HBH Options, UDP Options, etc.

It's slightly different incentive logic, though.  With HbH, the entire path, including transit networks, has to not bork the HbH option.  The sender, the client, and the client's network (Wi-Fi, 5G RAN, cablemodem) has an incentive for the additional network-to-host signaling.  But the transit network doesn't care and the transit networks, today, actively harm HbH options.

Tunneling over those harmful transit networks using something like MASQUE (reverse MASQUE, perhaps), might be a way to get HbH across them to those cooperating ISPs.  Or just avoid transit networks which all the big players do anyways.  My company and our customers are not big players and cannot avoid transit networks.

> 
> 
>> The "don't care about UDP trailers" behavior of transit networks contrasts from IPv6 HbH options where the transit networks are, today, interfering with IPv6 HbH options, and do not have an aligned incentive to change their IPv6 HbH operations.
> 
> 
> Probing can be done for HBH Options as well (see draft). There are some differences. UDP Options probing is for establishing feasibility across the path as well as the end host, HBH options probing and Happy Eyeballs are really only probing the path. One likely difference in practice is that UDP Options probing probably needs to be done on a 4-tuple basis because the end host might have an Anycast address that is behind a load balancer (might be a minor point, but will require testing and consideration to implement a robust probing algorithm).
> 
> Without any deployment of UDP Options and not a lot of implementations, I'm quite skeptical of any statement that UDP Options are any more survivable than HBH Options. We don't really know. And, even if it were true today, there's a very real possibility the providers will block them if they sense the slightest security issue, especially if they can't parse these because they are trailers (i.e. trailers might be perceived to be a covert channel to firewalls that can't process trailers).

We only have https://dl.ifip.org/db/conf/tma/tma2020/tma2020-camera-paper70.pdf and the hope that transit providers aren't doing (broken) checksum validation and that the eyeball network operators (ISPs) are interested in helping their customers.

-d