Re: [Masque] MASQUE extensibility

[Mirja Kuehlewind <[e-mail]>]

Hi all,

what David described as repeatable or non-repeatable data, I would rather distinguish based on the fact if the information is need per packet or per flow. For ECN there are two approaches to provide the information one or the other way (of course enabling a different granularity):


  1.  You can provide the information per packet by indicating for all packets from the client the IP ECN codepoint that should be set on outgoing packets and for all packet to the client which IP ECN codepoint was set when received by the proxy. This information is needed for each and every packet (of course you can define some default but that might not safe much) and provides the client the most information. So you can simply request support with the CONNECT-UDP request and prefix very datagram/stream frame with a field.
  2.  You can also just provide ECN feedback based one accumulated counters similar as QUIC ECN feedback is provided anyway and also how Accurate ECN works. Given this is the same granularity as provided in the transport feedback to the sender, this feedback granularity is sufficient for congestion control. However, for setting the ECN IP codepoint on outgoing packet the client might want to specify more specifically which codepoint should be set on which packet, e.g. for probing. However, there are ways to realize that as well, e.g. probing could actually also be done by the proxy and reported to the client, or the client could have defined point where the codepoint can be changed in a flow. Anyway this kind of signal would not be specifically related to a certain packet and could be done based on a request-reply scheme between the client and the proxy on the stream that has been used to request the CONNECT-UDP.

I’m not sure yet which is the better design for ECN, however, not matter what we decide for ECN, I believe we need both signaling scheme for different uses.

Mirja




From: Martin Duke <[e-mail]>
Date: Thursday, 7. January 2021 at 00:09
To: David Schinazi <[e-mail]>
Cc: Ben Schwartz <[e-mail]>, Christopher Wood <[e-mail]>, Mirja Kuehlewind <[e-mail]>, "[e-mail]" <[e-mail]>
Subject: Re: [Masque] MASQUE extensibility

I'm increasingly optimistic that this will work. For a while I was concerned that including DSCP + ECN would use up 8 bits, but in practice you'd use no more than a handful of codepoints and:
1) As David suggests, the extension can always define that a flow ID contains some metadata before the payload
2) In the method you can ad hoc define some flow IDs to correspond to the DSCPs you want to use: eg, Datagram-Flow-Id = 42; AF13, 44; AF23, 46; other [other would be used by the proxy if it receives something besides AF13/AF23 from the server]

On Wed, Jan 6, 2021 at 2:28 PM David Schinazi <[e-mail]<mailto:[e-mail]>> wrote:
+1 to what Martin said, WebRTC can be supported by making multiple CONNECT-UDP requests.

Regarding Ben's example of many bits, I'd regard that as what I called non-repeatable-metadata earlier. Let's say that you want to build an extension to CONNECT-UDP that optionally encodes a 32bit timestamp, some datagrams carry a timestamp and some don't. You would send:
    Datagram-Flow-Id = 42, 44; timestamp
This would have the following semantics:
- all datagrams with flow ID 42 just contain the UDP payload
- all datagrams with flow ID 44 contain a 32bit timestamp followed by the UDP payload

I'm not suggesting to encode all metadata in the flow ID, only metadata that is very often repeated like ECN bits. If you have a large number of bits, then Ben's right than encoding it in the flow ID is a bad idea because it would use up 2^N flow IDs. I'm suggesting to use flow IDs for repeatable-metadata, and the payload for non-repeatable-metadata. Sorry if this wan't clear from my earlier message.

David


On Wed, Jan 6, 2021 at 11:18 PM Martin Duke <[e-mail]<mailto:[e-mail]>> wrote:
I'm not sure I understand, Ben.

I imagine binding a client port to each of those endpoints is a separate UDP-CONNECT exchange -- they are different IP flows. Or are you asking for a "UDP Listen" where the peer IP/Port is unknown?

On Wed, Jan 6, 2021 at 8:15 AM Ben Schwartz <[e-mail]<mailto:[e-mail]>> wrote:
One important use case that I think is missing here is WebRTC (i.e. ICE).  For ICE to work efficiently, the user needs to be able to bind a single "client port" and use it to communicate with several (~3-10) distinct endpoints.  These endpoints are typically not known until _after_ connection setup.

At a minimum, this would require a way to add new flow IDs, bound to distinct destinations, after the tunnel is up.  That would exactly parallel TURN's ChannelBind request.  This is possible within the current CONNECT-UDP draft's framework.

My main concern with using flow IDs for metadata is that it invites a combinatorial explosion: encoding N bits of metadata requires allocating O(2^N) flow IDs in the Datagram-Flow-Id header.  With 2 bits of ECN, we're OK, but it's not hard to see how we could get to 10+ bits over time.  However, if flow IDs can be allocated as needed via new commands in Stream Chunks, this makes the problem much less pressing (only combinations actually used must be allocated).

Regarding lifetimes, cleaning up unused flow IDs can also be handled within the current framework, by adding another new command in a Stream Chunk.

On Wed, Jan 6, 2021 at 9:31 AM David Schinazi <[e-mail]<mailto:[e-mail]>> wrote:
Thanks! Yes, that's exactly what I meant by "repeatable" - perhaps "repeated" would have been a better choice of word here.

David

On Wed, Jan 6, 2021 at 3:26 PM Christopher Wood <[e-mail]<mailto:[e-mail]>> wrote:
On Wed, Jan 6, 2021, at 1:06 AM, David Schinazi wrote:
> Hi Chris,
>
> Thanks for your note, I think that we should strive to reach consensus
> on extensibility goals, requirements, and means around our upcoming
> interim. However, I don't understand all of your individual questions
> (for example, I don't understand what the word "fixed" means in the
> first question). Perhaps allow me to try to phrase these questions
> differently, to attempt to distill fundamental concepts.
>
> What we're mainly building is a way to convey data (e.g. the payload of
> a UDP packet) and metadata (e.g. the UDP port number); I'll further
> subdivide metadata into two categories: repeatable-metadata (e.g. the
> IP address of the target-server, ECN-ECT(0) vs CE) and
> non-repeatable-metadata (e.g. the timestamp of when the UDP packet was
> received). One way we could have built CONNECT-UDP was to send all
> repeatable-metadata and non-repeatable-metadata in every single packet,
> i.e. every single DATAGRAM frame carries the target-server IP and port.
> But, in order to improve efficiency and simplify management on the
> proxy, we decided to associate repeatable-metadata with a flow ID, and
> then only transmit the flow ID in DATAGRAM frames. This introduced the
> need for managing lifetimes (how long does the proxy maintain the
> mapping between flow ID and target-server IP+port?). We decided to tie
> this lifetime to the existence of the bidirectional stream that carried
> the CONNECT-UDP request. We did this because it resembled how CONNECT
> worked, but also because it was efficient and relatively simple to
> implement.

Am I correct in assuming that "repeatable" here means "possibly sent more than once across different packets"? I assume so, given that the target address is one such example.

Either way, this is a *much better* phrasing of the fundamental problem. Thanks for writing it up. :-)

> Now, to dive into your questions, let's focus on the example of
> CONNECT-UDP with ECN support. ECN adds two more bits of metadata per
> packet, and those bits are repeatable-metadata. From my point of view,
> the most logical way to encode those is to reuse the mechanism we built
> for vanilla CONNECT-UDP: we associate this repeatable-metadata with a
> flow ID. The latest version of the drafts (see links below) does this
> by having the CONNECT-UDP request/response negotiate four flow IDs. But
> we could have built this differently.
>
> Question 1: should repeatable-metadata be sent in each DATAGRAM frame,
> or associated with something like a flow ID?
>     The efficiency benefits lead me to conclude that associated is
> preferable.
>
> Question 2: should the lifetimes of these mapping be distinct or tied?
> I.e., should we be able to get in a state where we have a mapping for
> ECN-CE but not for ECN-ECT(0)?
>     Separate lifetimes can lead to odd bugs, so I'd prefer to tie them.
> This means not using four bidirectional streams per CONNECT-UDP with
> ECN flow.
>
> Question 3: Where do we encode this mapping? Flow ID or new QUIC or H3
> frames?
>     The flow ID is a simple concept that is simple to implement,
> especially if you don't control the underlying QUIC stack, so I prefer
> those.
>
> With this reasoning, I think that the current proposal is a good
> solution to our constraints. But if someone disagrees with these
> conclusions, please let me know - perhaps there are other concerns that
> I'm not aware of.
>
> draft-schinazi-masque-h3-datagram-04
> <https://tools.ietf.org/html/draft-schinazi-masque-h3-datagram-04>
> draft-ietf-masque-connect-udp-03
> <https://tools.ietf.org/html/draft-ietf-masque-connect-udp-03>
> draft-schinazi-masque-connect-udp-ecn-01
> <https://tools.ietf.org/html/draft-schinazi-masque-connect-udp-ecn-01>
>
> Cheers,
> David
>
> On Tue, Jan 5, 2021 at 9:05 PM Christopher Wood <[e-mail]<mailto:[e-mail]>> wrote:
> > Happy New Year, all!
> >
> > Thanks to everyone who chimed in on this thread! To summarize responses we received so far, it seems clear that we want extensibility for CONNECT-UDP and the future IP proxying mechanism, and that these should exist in separate documents. David’s draft specifying an ECN extension [1] is one such example. It also seems clear that HTTP headers make for a perfectly fine extension point for CONNECT-X requests.
> >
> > What remains unclear is how to support extensions within a proxy context, i.e., a CONNECT-UDP stream. Martin nicely outlined some of the variants, such as encoding things in per-datagram framing bits, negotiating and encoding information in flow IDs, or using different frame types entirely. We should try to converge on which of these options we need for generic tunneling or proxying use cases.
> >
> > To that end, here’s a couple questions that might help us get there:
> >
> > - Should flow ID interpretation be negotiated as part of an extension, or should it be fixed for CONNECT-UDP? (The latter is the approach taken in [1].)
> > - What are the tradeoffs between per-datagram signalling state versus, say, a different datagram frame type for different use cases? And which of these is maximally useful for CONNECT-UDP use cases?
> > - What are the tradeoffs between per-datagram and flow ID signalling?
> >
> > We hope these (or related questions) can be discussed before and during the interim.
> >
> > Thanks,
> > Chris and Eric
> >
> > [1] https://tools.ietf.org/html/draft-schinazi-masque-connect-udp-ecn-00
> >
> > On Wed, Dec 9, 2020, at 12:03 PM, Martin Duke wrote:
> > > Thanks for starting this discussion, Chris.
> > >
> > > With my AD hat on, I would personally interpret the charter to allow
> > > CONNECT-UDP to provide all the functionality one might have in a UDP
> > > socket, except those (like multicast) explicitly forbidden. This
> > > includes ECN, DSCP, etc, although the WG is welcome to decide that some
> > > or all of these functions aren't important. As AD, I don't have a
> > > strong opinion today as to whether these end up as one draft or
> > > multiple drafts. Certainly, it makes sense to spell out the design in a
> > > separate draft first and later decide whether or not to integrate it
> > > into the base design, as quicwg did with the spin bit and ECN support.
> > >
> > > If the CONNECT-UDP method is designed so as to be literally not
> > > extensible to support these use cases, I would consider that a
> > > significant flaw. I don't think that's the case.
> > >
> > > Speaking as an individual, the hardest extensibility problem would seem
> > > to be per-datagram information. There are four different resources we
> > > could use for this:
> > > - an additional QUIC frame type codepoint, with the second type having
> > > additional per-datagram information
> > > - an additional H3 frame type codepoint, with the second type having
> > > additional per-datagram information
> > > - flow ID codepoints, as David suggests. If we want to support DSCP,
> > > this means we need 8 bits per flow just for this (but then, we have
> > > 2^62 of them!)
> > > - Make every H3 datagram frame encode this information -- if we were to
> > > choose this option, we will probably end up using more H3 frame types
> > > as Webtransport, etc. will probably not want the overhead.
> > >
> > > To me, the first two are the least profligate, and the second option
> > > (another H3 frame) is the right layer to do it. But none of these
> > > resources are scarce and this is mainly an aesthetic preference.
> > >
> > > Broadening from the per-datagram issue, we are likely filling up some
> > > sort of HTTP or H3 registry with whatever extension points we specify,
> > > so that community's opinion is valuable here.
> > >
> > > Martin
> > >
> > > On Wed, Dec 9, 2020 at 10:01 AM Mirja Kuehlewind
> > > <mirja.kuehlewind=[e-mail]<mailto:[e-mail]>> wrote:
> > > > 1) Yes, every new protocol should be extensible. Lucas and others mentioned that H3 provides already some extensibility, however, as soon as you have send the CONNECT there is not much H3 "left" and therefore we should make sure that we consider options for additional extension mechanisms later in the life time of a CONNECT request.
> > > >
> > > > 2) People did mention in-stream (or I would say in-forwarding-association - I  think we need to agree on terminology here to make sure we talk about the same thing) control data. So I think there is information that you want to signal that is associated to a forwarding stream or even a certain packet, where it probably make sense to send this data within the respective forwarding stream, however, there might be other cases where you want to signal something based on some event in the proxy, including negotiation when or before the CONNECT is sent, or general information about proxy state or capabilities which maybe be independent of any active forwarding association. We also need to consider appropriate ways to signal such information and I think all (new) signaling we define should be extensible.
> > > >
> > > > 3) I think the extension points itself need to be described in the base spec. I also still think that ECN handling should be part of the base spec as this is an important functionality that is in my view not optional if we ever want to see it used. However we can of course start with separate documents and merge in when ready. Still, we first must agree about the kind of extension points we want to use.
> > > >
> > > > Mirja
> > > >
> > > >
> > > > On 04.12.20, 21:19, "Masque on behalf of Christopher Wood" <[e-mail]<mailto:[e-mail]> on behalf of [e-mail]<mailto:[e-mail]>> wrote:
> > > >
> > > >     One point raised during our IETF 109 meeting was extensibility. A number of different extensions have already been discussed, including: IP header compression, ECN signals, and QUIC-aware proxy support. It seems clear that the MASQUE use-cases require some amount of extensibility in the core mechanism. What we’d like to determine is how much extensibility is needed.
> > > >
> > > >     Given that our charter is somewhat vague on what extensions are in scope [1], there seem to be (at least) three questions we should answer if we are to embark on this extension work:
> > > >
> > > >     1) Do we want CONNECT-UDP and a future mechanism for IP proxying to be extensible?
> > > >     2) If yes, which parts of each protocol need extensibility, and to what extent?
> > > >     3) If yes, do we want to adopt these extensions as distinct documents? (An ECN signal, for example, might be included as an extension in the CONNECT-UDP document, or it might be part of a separate document. Conversely, extensions for QUIC-aware proxying seem likely to be a separate document.)
> > > >
> > > >     We'd like to hear answers to these three questions from the WG. Converging here will help us better determine what is in scope going forward. To that end, please share your thoughts on these questions before December 18. We'll assess and see where we are at that time.
> > > >
> > > >     Thanks,
> > > >     Chris and Eric
> > > >
> > > >     [1] Relevant text in the charter (https://datatracker.ietf.org/doc/charter-ietf-masque/) reads as follows:
> > > >
> > > >     The primary goal of this working group is to develop mechanism(s) that allow configuring and concurrently running multiple proxied stream- and datagram-based flows inside an HTTPS connection. These mechanism(s) are collectively called MASQUE. ***The group will specify HTTP and/or HTTP/3 extensions to enable this functionality.***
> > > >
> > > >     (emphasis ours)
> > > >
> > > >     --
> > > >     Masque mailing list
> > > >     [e-mail]<mailto:[e-mail]>
> > > >     https://www.ietf.org/mailman/listinfo/masque
> > > >
> > > > --
> > > > Masque mailing list
> > > > [e-mail]<mailto:[e-mail]>
> > > > https://www.ietf.org/mailman/listinfo/masque
> >
> > --
> > Masque mailing list
> > [e-mail]<mailto:[e-mail]>
> > https://www.ietf.org/mailman/listinfo/masque
--
Masque mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/masque