Re: [AVTCORE] Side meeting on SFrame over RTP

[Jonathan Lennox <[e-mail]>]

> On Nov 9, 2022, at 12:33 PM, Jonathan Lennox <[e-mail]> wrote:
> 
> Inspired by the discussion at the AVTCore meeting yesterday, we will have a side meeting this afternoon from 15:00-16:00 GMT (apologies for the short notice) in Mezzanine 12 (East Tower of the Metropole) on issues related to SFrame over RTP.

A small group of us (me, Magnus Westerlund, Martin Thomson, Richard Barnes, Youenn Fablet, and Jörg Ott) met this afternoon, and had what I think was a very productive discussion.

No one was taking notes contemporaneously, but I’ll try to summarize the conclusions we came to.  (If I got something wrong or left something out, I invite the other participants to send their own recollection as well!)

Youenn took the action to update draft-murillo-avtcore-multi-codec-payload-format to describe this solution.  He’d welcome volunteers to help him, though.


Our conclusion was that the basic approaches from the payload formats previously proposed for SFrame were good, but it would be better to describe them explicitly as a payload format for SFrame, to make it clear what receivers should do with them (i.e., perform the e2e auth/decryption process).

The SFrame’s encrypted data will encapsulate RTP payloads (not raw elementary stream payloads) — i.e. this will include the “payload header" data that incorporates fields like the VP8 and VP9 picture ID, layer IDs, and tl0picidx fields.  These are fields that the receiver process needs in order to know which frames it can safely hand off to a decoder process in such that it can decode the frame properly, vs. e.g. buffering them to wait for retransmissions or other repair packets.

However, this payload can (optionally) be treated as though it had “infinite” MTU, encompassing an entire encoded frame into one RTP payload before the SFrame encapsulation.  The SFrame is then (if you’re sending over UDP, as opposed to QUIC) fragmented post-encryption, mapping it over multiple RTP packets. (This lets you amortize the SFrame overhead across all the packets of the frame.). This fragmentation is done outside the e2e context, so an SFU can safely split or reassemble fragments if gatewaying between RTP-over-UDP and RTP-over-QUIC.

Alternatively, as an implementation choice, an encoder can produce MTU-targeted payloads, such that each payload is separately encapsulated in an SFrame.  This allows packets to be independently decodable, at the cost of paying the SFrame overhead per-packet.  (This is the mode that’s previously been described as “SPacket”; it could be advisable, e.g., for H.264 slices, if you believe you’re in an environment where decoders can usefully decode individual slices of a frame.)

The RTP header for these fragmented packets would carry the same information as the header would have on a non-SFrame packet, except for the payload type and possibly the marker bit.  The payload type would be a payload that indicates that this is an SFrame packet.  Any SFU header extension information (frame marking, or AV1 dependency descriptor) would be copied into the RTP packets.
 
We discussed whether it would be better to define payload types as, e.g. “VP8 over SFrame”, “VP9 over SFrame”, “AV1 over SFrame” (which could be thought of as “rtx-style”), or rather just have an “SFrame” payload type which carries its inner payload type as a field inside it (“red-style” or “flexfec-style”).  Either approach would work, but given that we’re already frequently hitting issues in WebRTC with payload type exhaustion, the latter seemed like it would be more practical.  But this can be something the WG decides.
 
That said, even in the latter case this field should probably *not* be part of the e2e protected data, because there are cases where due to negotiation oddities SFUs need to be able to rewrite payload type numbers.
 
For the marker bit, we may want to use the marker bit to mean “end of fragmented sframe” rather than “end of frame”, and put the original marker bit somewhere in the SFrame payload, but this is something that can be figured out as part of the bit details of the SFrame payload format.