Re: [Masque] MASQUE extensibility

[Martin Duke <[e-mail]>]

I'm not sure I understand, Ben.

I imagine binding a client port to each of those endpoints is a separate
UDP-CONNECT exchange -- they are different IP flows. Or are you asking for
a "UDP Listen" where the peer IP/Port is unknown?

On Wed, Jan 6, 2021 at 8:15 AM Ben Schwartz <[e-mail]> wrote:

> One important use case that I think is missing here is WebRTC (i.e. ICE).
> For ICE to work efficiently, the user needs to be able to bind a single
> "client port" and use it to communicate with several (~3-10) distinct
> endpoints.  These endpoints are typically not known until _after_
> connection setup.
>
> At a minimum, this would require a way to add new flow IDs, bound to
> distinct destinations, after the tunnel is up.  That would exactly parallel
> TURN's ChannelBind request.  This is possible within the current
> CONNECT-UDP draft's framework.
>
> My main concern with using flow IDs for metadata is that it invites a
> combinatorial explosion: encoding N bits of metadata requires allocating
> O(2^N) flow IDs in the Datagram-Flow-Id header.  With 2 bits of ECN, we're
> OK, but it's not hard to see how we could get to 10+ bits over time.
> However, if flow IDs can be allocated as needed via new commands in Stream
> Chunks, this makes the problem much less pressing (only combinations
> actually used must be allocated).
>
> Regarding lifetimes, cleaning up unused flow IDs can also be handled
> within the current framework, by adding another new command in a Stream
> Chunk.
>
> On Wed, Jan 6, 2021 at 9:31 AM David Schinazi <[e-mail]>
> wrote:
>
>> Thanks! Yes, that's exactly what I meant by "repeatable" - perhaps
>> "repeated" would have been a better choice of word here.
>>
>> David
>>
>> On Wed, Jan 6, 2021 at 3:26 PM Christopher Wood <[e-mail]>
>> wrote:
>>
>>> On Wed, Jan 6, 2021, at 1:06 AM, David Schinazi wrote:
>>> > Hi Chris,
>>> >
>>> > Thanks for your note, I think that we should strive to reach consensus
>>> > on extensibility goals, requirements, and means around our upcoming
>>> > interim. However, I don't understand all of your individual questions
>>> > (for example, I don't understand what the word "fixed" means in the
>>> > first question). Perhaps allow me to try to phrase these questions
>>> > differently, to attempt to distill fundamental concepts.
>>> >
>>> > What we're mainly building is a way to convey data (e.g. the payload
>>> of
>>> > a UDP packet) and metadata (e.g. the UDP port number); I'll further
>>> > subdivide metadata into two categories: repeatable-metadata (e.g. the
>>> > IP address of the target-server, ECN-ECT(0) vs CE) and
>>> > non-repeatable-metadata (e.g. the timestamp of when the UDP packet was
>>> > received). One way we could have built CONNECT-UDP was to send all
>>> > repeatable-metadata and non-repeatable-metadata in every single
>>> packet,
>>> > i.e. every single DATAGRAM frame carries the target-server IP and
>>> port.
>>> > But, in order to improve efficiency and simplify management on the
>>> > proxy, we decided to associate repeatable-metadata with a flow ID, and
>>> > then only transmit the flow ID in DATAGRAM frames. This introduced the
>>> > need for managing lifetimes (how long does the proxy maintain the
>>> > mapping between flow ID and target-server IP+port?). We decided to tie
>>> > this lifetime to the existence of the bidirectional stream that
>>> carried
>>> > the CONNECT-UDP request. We did this because it resembled how CONNECT
>>> > worked, but also because it was efficient and relatively simple to
>>> > implement.
>>>
>>> Am I correct in assuming that "repeatable" here means "possibly sent
>>> more than once across different packets"? I assume so, given that the
>>> target address is one such example.
>>>
>>> Either way, this is a *much better* phrasing of the fundamental problem.
>>> Thanks for writing it up. :-)
>>>
>>> > Now, to dive into your questions, let's focus on the example of
>>> > CONNECT-UDP with ECN support. ECN adds two more bits of metadata per
>>> > packet, and those bits are repeatable-metadata. From my point of view,
>>> > the most logical way to encode those is to reuse the mechanism we
>>> built
>>> > for vanilla CONNECT-UDP: we associate this repeatable-metadata with a
>>> > flow ID. The latest version of the drafts (see links below) does this
>>> > by having the CONNECT-UDP request/response negotiate four flow IDs.
>>> But
>>> > we could have built this differently.
>>> >
>>> > Question 1: should repeatable-metadata be sent in each DATAGRAM frame,
>>> > or associated with something like a flow ID?
>>> >     The efficiency benefits lead me to conclude that associated is
>>> > preferable.
>>> >
>>> > Question 2: should the lifetimes of these mapping be distinct or tied?
>>> > I.e., should we be able to get in a state where we have a mapping for
>>> > ECN-CE but not for ECN-ECT(0)?
>>> >     Separate lifetimes can lead to odd bugs, so I'd prefer to tie
>>> them.
>>> > This means not using four bidirectional streams per CONNECT-UDP with
>>> > ECN flow.
>>> >
>>> > Question 3: Where do we encode this mapping? Flow ID or new QUIC or H3
>>> > frames?
>>> >     The flow ID is a simple concept that is simple to implement,
>>> > especially if you don't control the underlying QUIC stack, so I prefer
>>> > those.
>>> >
>>> > With this reasoning, I think that the current proposal is a good
>>> > solution to our constraints. But if someone disagrees with these
>>> > conclusions, please let me know - perhaps there are other concerns
>>> that
>>> > I'm not aware of.
>>> >
>>> > draft-schinazi-masque-h3-datagram-04
>>> > <https://tools.ietf.org/html/draft-schinazi-masque-h3-datagram-04>
>>> > draft-ietf-masque-connect-udp-03
>>> > <https://tools.ietf.org/html/draft-ietf-masque-connect-udp-03>
>>> > draft-schinazi-masque-connect-udp-ecn-01
>>> > <https://tools.ietf.org/html/draft-schinazi-masque-connect-udp-ecn-01>
>>> >
>>> > Cheers,
>>> > David
>>> >
>>> > On Tue, Jan 5, 2021 at 9:05 PM Christopher Wood <[e-mail]>
>>> wrote:
>>> > > Happy New Year, all!
>>> > >
>>> > > Thanks to everyone who chimed in on this thread! To summarize
>>> responses we received so far, it seems clear that we want extensibility for
>>> CONNECT-UDP and the future IP proxying mechanism, and that these should
>>> exist in separate documents. David’s draft specifying an ECN extension [1]
>>> is one such example. It also seems clear that HTTP headers make for a
>>> perfectly fine extension point for CONNECT-X requests.
>>> > >
>>> > > What remains unclear is how to support extensions within a proxy
>>> context, i.e., a CONNECT-UDP stream. Martin nicely outlined some of the
>>> variants, such as encoding things in per-datagram framing bits, negotiating
>>> and encoding information in flow IDs, or using different frame types
>>> entirely. We should try to converge on which of these options we need for
>>> generic tunneling or proxying use cases.
>>> > >
>>> > > To that end, here’s a couple questions that might help us get there:
>>> > >
>>> > > - Should flow ID interpretation be negotiated as part of an
>>> extension, or should it be fixed for CONNECT-UDP? (The latter is the
>>> approach taken in [1].)
>>> > > - What are the tradeoffs between per-datagram signalling state
>>> versus, say, a different datagram frame type for different use cases? And
>>> which of these is maximally useful for CONNECT-UDP use cases?
>>> > > - What are the tradeoffs between per-datagram and flow ID signalling?
>>> > >
>>> > > We hope these (or related questions) can be discussed before and
>>> during the interim.
>>> > >
>>> > > Thanks,
>>> > > Chris and Eric
>>> > >
>>> > > [1]
>>> https://tools.ietf.org/html/draft-schinazi-masque-connect-udp-ecn-00
>>> > >
>>> > > On Wed, Dec 9, 2020, at 12:03 PM, Martin Duke wrote:
>>> > > > Thanks for starting this discussion, Chris.
>>> > > >
>>> > > > With my AD hat on, I would personally interpret the charter to
>>> allow
>>> > > > CONNECT-UDP to provide all the functionality one might have in a
>>> UDP
>>> > > > socket, except those (like multicast) explicitly forbidden. This
>>> > > > includes ECN, DSCP, etc, although the WG is welcome to decide that
>>> some
>>> > > > or all of these functions aren't important. As AD, I don't have a
>>> > > > strong opinion today as to whether these end up as one draft or
>>> > > > multiple drafts. Certainly, it makes sense to spell out the design
>>> in a
>>> > > > separate draft first and later decide whether or not to integrate
>>> it
>>> > > > into the base design, as quicwg did with the spin bit and ECN
>>> support.
>>> > > >
>>> > > > If the CONNECT-UDP method is designed so as to be literally not
>>> > > > extensible to support these use cases, I would consider that a
>>> > > > significant flaw. I don't think that's the case.
>>> > > >
>>> > > > Speaking as an individual, the hardest extensibility problem would
>>> seem
>>> > > > to be per-datagram information. There are four different resources
>>> we
>>> > > > could use for this:
>>> > > > - an additional QUIC frame type codepoint, with the second type
>>> having
>>> > > > additional per-datagram information
>>> > > > - an additional H3 frame type codepoint, with the second type
>>> having
>>> > > > additional per-datagram information
>>> > > > - flow ID codepoints, as David suggests. If we want to support
>>> DSCP,
>>> > > > this means we need 8 bits per flow just for this (but then, we
>>> have
>>> > > > 2^62 of them!)
>>> > > > - Make every H3 datagram frame encode this information -- if we
>>> were to
>>> > > > choose this option, we will probably end up using more H3 frame
>>> types
>>> > > > as Webtransport, etc. will probably not want the overhead.
>>> > > >
>>> > > > To me, the first two are the least profligate, and the second
>>> option
>>> > > > (another H3 frame) is the right layer to do it. But none of these
>>> > > > resources are scarce and this is mainly an aesthetic preference.
>>> > > >
>>> > > > Broadening from the per-datagram issue, we are likely filling up
>>> some
>>> > > > sort of HTTP or H3 registry with whatever extension points we
>>> specify,
>>> > > > so that community's opinion is valuable here.
>>> > > >
>>> > > > Martin
>>> > > >
>>> > > > On Wed, Dec 9, 2020 at 10:01 AM Mirja Kuehlewind
>>> > > > <mirja.kuehlewind=[e-mail]> wrote:
>>> > > > > 1) Yes, every new protocol should be extensible. Lucas and
>>> others mentioned that H3 provides already some extensibility, however, as
>>> soon as you have send the CONNECT there is not much H3 "left" and therefore
>>> we should make sure that we consider options for additional extension
>>> mechanisms later in the life time of a CONNECT request.
>>> > > > >
>>> > > > > 2) People did mention in-stream (or I would say
>>> in-forwarding-association - I  think we need to agree on terminology here
>>> to make sure we talk about the same thing) control data. So I think there
>>> is information that you want to signal that is associated to a forwarding
>>> stream or even a certain packet, where it probably make sense to send this
>>> data within the respective forwarding stream, however, there might be other
>>> cases where you want to signal something based on some event in the proxy,
>>> including negotiation when or before the CONNECT is sent, or general
>>> information about proxy state or capabilities which maybe be independent of
>>> any active forwarding association. We also need to consider appropriate
>>> ways to signal such information and I think all (new) signaling we define
>>> should be extensible.
>>> > > > >
>>> > > > > 3) I think the extension points itself need to be described in
>>> the base spec. I also still think that ECN handling should be part of the
>>> base spec as this is an important functionality that is in my view not
>>> optional if we ever want to see it used. However we can of course start
>>> with separate documents and merge in when ready. Still, we first must agree
>>> about the kind of extension points we want to use.
>>> > > > >
>>> > > > > Mirja
>>> > > > >
>>> > > > >
>>> > > > > On 04.12.20, 21:19, "Masque on behalf of Christopher Wood" <
>>> [e-mail] on behalf of [e-mail]> wrote:
>>> > > > >
>>> > > > >     One point raised during our IETF 109 meeting was
>>> extensibility. A number of different extensions have already been
>>> discussed, including: IP header compression, ECN signals, and QUIC-aware
>>> proxy support. It seems clear that the MASQUE use-cases require some amount
>>> of extensibility in the core mechanism. What we’d like to determine is how
>>> much extensibility is needed.
>>> > > > >
>>> > > > >     Given that our charter is somewhat vague on what extensions
>>> are in scope [1], there seem to be (at least) three questions we should
>>> answer if we are to embark on this extension work:
>>> > > > >
>>> > > > >     1) Do we want CONNECT-UDP and a future mechanism for IP
>>> proxying to be extensible?
>>> > > > >     2) If yes, which parts of each protocol need extensibility,
>>> and to what extent?
>>> > > > >     3) If yes, do we want to adopt these extensions as distinct
>>> documents? (An ECN signal, for example, might be included as an extension
>>> in the CONNECT-UDP document, or it might be part of a separate document.
>>> Conversely, extensions for QUIC-aware proxying seem likely to be a separate
>>> document.)
>>> > > > >
>>> > > > >     We'd like to hear answers to these three questions from the
>>> WG. Converging here will help us better determine what is in scope going
>>> forward. To that end, please share your thoughts on these questions before
>>> December 18. We'll assess and see where we are at that time.
>>> > > > >
>>> > > > >     Thanks,
>>> > > > >     Chris and Eric
>>> > > > >
>>> > > > >     [1] Relevant text in the charter (
>>> https://datatracker.ietf.org/doc/charter-ietf-masque/) reads as follows:
>>> > > > >
>>> > > > >     The primary goal of this working group is to develop
>>> mechanism(s) that allow configuring and concurrently running multiple
>>> proxied stream- and datagram-based flows inside an HTTPS connection. These
>>> mechanism(s) are collectively called MASQUE. ***The group will specify HTTP
>>> and/or HTTP/3 extensions to enable this functionality.***
>>> > > > >
>>> > > > >     (emphasis ours)
>>> > > > >
>>> > > > >     --
>>> > > > >     Masque mailing list
>>> > > > >     [e-mail]
>>> > > > >     https://www.ietf.org/mailman/listinfo/masque
>>> > > > >
>>> > > > > --
>>> > > > > Masque mailing list
>>> > > > > [e-mail]
>>> > > > > https://www.ietf.org/mailman/listinfo/masque
>>> > >
>>> > > --
>>> > > Masque mailing list
>>> > > [e-mail]
>>> > > https://www.ietf.org/mailman/listinfo/masque
>>>
>> --
>> Masque mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/masque
>>
>