[Sframe] A possible framework for SFrame and repacketization

[Richard Barnes <[e-mail]>]

Hey AVTCORE and SFrame WGs,

After the SFrame / repacketization discussion today, I had a few more
thoughts on this question, which I think might point toward a productive
direction here.  The below notes are a little stream-of-consciousness, but
hopefully get the point across.

tl;dr: We should treat SFrame and generic fragmentation as new layers in
the RTP processing stack.  If we do that, the design and the work to do
become pretty clear and straightforward.

First: The repacketization problem in RTP is not SFrame-specific.  It
arises any time the media forwarder doesn't understand the codec, which
could be due to encryption or just software.  As Jonathan pointed out in
the room, RTP generally assumes that a media forwarder does understand the
codec, but there are situations such as E2EE where that might not be the
case.  The question here is whether we accommodate these situations, and if
so how.

In terms of SFrame-in-RTP, there's still not consensus on where SFrame goes
in the media processing stack:

* Capture / Render
* Encode / Decode
* [[ does SFrame go here? == SFrame, requires generic packetization ]]
* Packetize / Depacketize
* [[ or does SFrame go here? == SPacket, prevents repacketization ]]
* SRTP protect / unprotect

Repacketization on opaque data effectively requires that there be two
layers of fragmentation / reassembly, the sender's fragemntation and the
forwarder's.  The sender's fragmentation has to be preserved because the
fragments might not make sense if the forwarder combined them or broke them
apart (e.g., they might not authenticate).  The forwarder's fragmentation
is necessary to fit within the MTU.  Fortunately, I think this actually
leads us to a nicer, more unified architecture for integrating SFrame:

* Capture / Render
* Encode / Decode
* Codec-aware fragment / reassemble
* SFrame protect / unprotect
* Opaque fragment / reassemble
* SRTP protect / unprotect

This model includes both the SFrame and SPacket cases as special cases.
For SFrame, the codec-aware fragmentation is trivial and the opaque
fragmentation is non-trivial; for SPacket, the opaque packetization is
trivial and the codec-aware fragmentation is non-trivial.

It also points to how you would define and negotiate the various parts:

* Define two new mechanisms:
  1. SFrame as a generic protect/unprotect algorithm, and
  2. a generic fragmentation / reassembly protocol  (basically Peter's
proposal)
* Signal these independently in SDP, and independently of the codec etc,
since they're independent layers.
  * If you negotiate (1) but not (2), you get SPacket and no
repacketization.
  * If you negotiate (1) and (2), then you get SFrame and repacketization.
  * If you negotiate (2) and not (1), you get generic repacketization on
any codec.

Conveniently, the SFrame WG is already working on (1); (2) would be
straightforward for AVTCORE to handle.

There are some details to work out, like how the RTP header metadata gets
associated across all these layers and how MTU responsibilities sort out
between the two fragmentation layers.  But maybe this is an overall
approach that makes sense to folks?

The only real alternative is the current state of affairs, where we only
have SPacket, no SFrame and no repacketization.  If we want either of
those, it seems like we need to do something like the above.

--Richard

P.S. For folks with the Encoded Transform discussion in mind, I think this
also provides a clean model for the "packet-level" processing we have been
discussing in that context.  You simply generalize the above stack so that
instead of "SFrame", it says "JS-provided transform".  The current
frame-level transforms would short-circuit the diagram, skipping the
codec-aware fragmentation stage.