Re: [Sframe] [dispatch] Magnus Westerlund's Block on charter-ietf-sframe-00-00: (with BLOCK and COMMENT)

[Bernard Aboba <[e-mail]>]

Sergio said:

"No one has spoken of putting everything on a single SSRC, the new RTP
packetization would change the way the payload is packetized into the
RTP packets, but not the rtp/ssrc/mid mapping of the media streams which
would be the same as if SFRAME is not in use. So this is a non-issue."

[BA] Right.  SFRAME doesn't change the SSRC allocation scheme, nor does it
prevent the SFU from modifying the SSRC.
If the application was sending simulcast with multiple SSRCs/RIDs, SFRAME
won't affect that.  Similarly with SVC, if the application was sending all
layers on a single SSRC, SFRAME won't change that either.

"If you read my example above, this information is known before the frame
goes to the Insertable Stream process, so it will be available for the RTP
packetization without going via SFRAME."

[BA] As you note, the metadata contains information about each frame, and
the Insertable Streams implementation assumes that the application
modifications don't invalidate the metadata, which currently contains info
such as frame type, frameId, dependencies, width, height, LID, TID, SSRC
and CSRCs.

Most of the RTP header extension fields are the same for each packet within
a frame; only a few will change between packets within a frame, such as the
Start and End bits.

The exercise is to make sure that the metadata is sufficient to enable the
generic RTP packetizer to fill in all the RTP header and RTP header
extension fields.  For the RTP header extension fields, this includes info
such as Discardable, Switch, Required, Base Layer Sync, etc.  Today it
isn't clear to me that this info is completely determinable from the
Insertable Streams metadata.

Given that we don't have a lot of deployment of frame forwarding RTP header
extensions yet, it's probably best to avoid tying SFRAME  to any particular
design.  For example, you wouldn't want a design that could only function
with the framemarking RTP header extension or the Generic Frame Descriptor
or the Dependency Descriptor, so that a new specification is needed when
the N+1 frame forwarding technology comes along.







On Mon, Sep 14, 2020 at 2:39 AM Sergio Garcia Murillo <
[e-mail]> wrote:

> On 14/09/2020 9:29, Magnus Westerlund wrote:
> > Hi,
> >
> > Please see inline
> >
> > On Fri, 2020-09-11 at 17:32 +0200, Sergio Garcia Murillo wrote:
> >> On 11/09/2020 13:19, Magnus Westerlund wrote:
> >>> Hi,
> >>>
> >>>>> So do I assume correctly that the idea is that the application layer
> >>>>> using SFRAME if it haves multiple independent or sets of dependent
> >>>>> streams there will be no support for that aspect in SFRAME layer?
> >>>>> Instead it is up to the application to put such information to either
> >>>>> put it inside the encapsualted part of the SFRAME or map it to the
> >>>>> lower transport layer, like to RTP SSRCs and extension information.
> >>>> If i understood you correctly, SFrame is kind of agnostic to the media
> >>>> streams. Currently it has a single global frame counter for all the
> >>> transport, so
> >>>> the number of streams/dependency between them is not known/needed
> >>>> by SFrame.
> >>> Yes, and when using a multi-stream application with layered encoding
> being
> >>> protected by SFRAME and transported over RTP with repair functions the
> high
> >>> layer application will need to have a model for how it maps individual
> >>> SFRAMEs to RTP layer functions to enable them to do their work. You
> get a
> >>> very limited functionality if in an SFU system tries to send all over a
> >>> single RTP SSRC. So this becomes a discussion in the RTP Payload format
> >>> context when carrying SFRAMEs.
> >>
> >> I don't really understand your point, let's put an example vp9 svc.
> >>
> >> Chrome will encode each picture from the video stream and pass it to the
> >> vp9 svc encoder. The encoder will produce n-frames (one per spatial
> >> layer) for a given picture.
> >>
> >> Chrome will get each frame with its metadata (spatial/layer id + layer
> >> structure info + previous frame dependencies + future frame depencies)
> >> and pass it over to the insertable streams.
> >>
> >> The payload will go to javascript, in where SFRAME will encrypt it and
> >> returned as a binary blob.
> >>
> >> The browser will get the binary blob, packetize it in several rtp
> >> packets according to the new packetization format and add the metadata
> >> as a header extension.
> >>
> >> Is this process what you are describing or is there anything missing?
> >>
> > Then the client will send these RTP packets to an SFU that will need to
> make
> > decision on which of the packets to forward for which layers to a
> particular
> > receiver. To be able to do this there are first the aspect of that the
> meta data
> > needs to be included so that the SFU can make decision based on the
> packets it
> > recieves. In RTP context this likely are a more generic structure like
> the RTP
> > header extension for Frame marking (draft-ietf-avtext-framemarking).
> >
> > For packet it doesn't receive it lacks the meta data. Thus, for a more
> efficient
> > repair of packets that are missing and to repair only packets that the
> SFU
> > actually needs, then you actually have to map the layered structure into
> RTP
> > level structures so that the SFU can determine that the missing packet
> belongs
> > to a layer that it actually intends to forward. Putting everything in a
> single
> > SSRC requires the SFU to repair all losses independent of it needing the
> packet
> > or not.
> >
> > A even more extreme example is if the client has 3 video cameras and
> produce 3
> > video encodings. In the SFRAME context it is possible to take the video
> frames
> > from all these threee encodings and put them in a single RTP stream
> (SSRC) with
> > SFRAMEs from that end-point. However, that would force your meta data to
> contain
> > also source identification rather than to using different SSRCs. Putting
> > multiple encoding on an single RTP stream would deprive the SFU of the
> > possibility to do rate control per encoding (RFC 5104 - TMMBR) as well
> as pause
> > streams it doesn't forward at all (RFC 7728 - Stream PAUSE) because all
> the
> > existing RTP mechanism that are included in WebRTC works on the RTP
> streams, not
> > sub-streams that doesn't have an identifier in the RTP layer. In
> addition to
> > having to repair any loss for the aggregated stream of the three
> encodings.
>
> No one has spoken of putting every on a single SSRC, the new rtp
> packetization would change the way the payload is packetized into the
> RTP packets, but not the rtp/ssrc/mid mapping of the media streams which
> would be the same as if not SFRAME is use. So this is a non-issue.
>
>
> >
> >>>>>> However, if the encrypted output is going to be transmitted over RTP
> >>>>>> using a new packetization format, we need to address how to
> negotiate
> >>>>>> that in the SDP.
> >>>>> Good
> >>>>>
> >>>>>> Note that this new packetization format will not only be alid for
> >>>>>> transporting encrypted frames (SFrame or not) but even any other
> >>>>>> audio/video frames.
> >>>>> I understand the potential exists. However, the recommendation for
> RTP
> >>>>> has been to try to consider Application Level Framing. In the case of
> >>>>> SFRAME that consideration moves up above SFRAME in the choice of how
> >>>>> data is split into SFRAMEs. However, having an RTP paylaod format for
> >>>>> SFRAME, then that should carry SFRAMEs. If you starts putting in
> other
> >>>>> binary objects into it, then you create a new demultiplexing point
> for
> >>>>> format between the RTP payload format and the SFRAME processing. That
> >>>> appears unmotivated.
> >>>>
> >>>>
> >>>> Note that SFRAME is not the only encryption possible with w3c
> insertable
> >>>> streams, it is up to the application to define its own crypto if they
> >>> want. So
> >>>> the payload must be able to transport non-SFRAME opaque blobs.
> >>> So I would consider this a misuse of the RTP payload format. The media
> type
> >>> will be for SFRAMEs. I understand that you might not be able to
> prevent this
> >>> in WebRTC. However, from an interoperability point of view you will be
> >>> stating that this is SFRAMEs. RTP will not be able to tell a
> difference. But
> >>> I don't see that this is will explicitly support carrying other things
> than
> >>> SFRAME. Because that means bringing in other consideration including
> the
> >>> security model for the E2E usage.
> >>
> >> Not sure if ignoring the reality on how the packetization format is
> >> going to be used within insertable streams is a good idea.
> > If you are passing other things than SFRAME you are also sending them
> without
> > end-to-end protection. Why is this happening, if the information was
> intended
> > for the SFU, then I think we should identify it as such, rather than
> having the
> > SFU have to hunt for it. If not, why are you sending it outside of the
> SFRAME
> > envelope?
>
> This is an application concern. I don't know why they would choose to do
> it, but the fact is that they can do it.
>
>
> >
> >>
> >>>>> Should the RTP payload format aspect of the charter be more explicit
> >>>>> in that it needs to disucss the general model of how to use SFRAME
> and
> >>>>> how an application can use the facilities of RTP to get good
> >>>>> performance from RTP mechanism like FEC and retransmission?
> >>>> I don't think so, RTX and FEC frames MUST not be modified/affected by
> >>>> SFRAME/the new packetization format, so they work out of the box. We
> can
> >>>> be explicit about that in the chapter as a requirement.
> >>>>
> >>> So I am not saying that RTX or FEC shall be modified. What I am saying
> is
> >>> that the SFRAME payload format should discuss the impact on the
> performance
> >>> of these functions depending on how you structure the SFRAMES across
> SSRCs.
> >>> The most simple example is the one where you put all layers for a video
> >>> encoding in a single SSRC. In such a stream you loose a single RTP
> packet
> >>> between the transmitting end-point and the SFU. Now the SFU is only
> >>> forwarding the base layer and not the enhancement layers. If base
> layer and
> >>> enhancement layer share RTP stream, the SFU can't determine if the
> missing
> >>> packet contains an base layer data or enhancement layer data. If the
> base
> >>> layer and enhancement layer would be on different SSRC, the fact that
> there
> >>> is a missing packet on the enhancement layer RTP stream means that the
> SFU
> >>> can ignore that as it anyway are not forwarding it. This same argument
> can
> >>> be applied between media sources. So how you map your application level
> >>> structures onto RTP do matter for the resulting performance of RTP.
> Putting
> >>> all on a single SSRC is not a path to good transport performance.
> >>
> >> But that is already the case for SVC codecs, so there is nothing new to
> >> specify in that regard.
> > So for SVC and other scalable codes there exists options for how to do
> this. And
> > where you can write and RTP packetizer for SVC that takes basicaly a
> mode and a
> > layer structure configuration as input, that is not possible for SFRAME
> as that
> > information is not visible in the RTP payload information. Instead it
> must come
> > with each SFRAME to packetize how to map it to the RTP layer.
> >
> > Thus, in my view the SFRAME RTP Payload format needs to discuss the core
> of
> > these issues to make it clear to the application both its options as
> well as
> > point to the impact of those options.
>
>
> This is already possible for any SVC codec re-using the Dependency
> Descriptor header extension:
>
> https://aomediacodec.github.io/av1-rtp-spec/#a1-introduction
>
> If you read my example above, this information is know before the frame
> goes to the Insertable Stream process, so it will be available for the
> rtp packetization without going via SFRAME.
>
>
> Best regards
>
> Sergio
>
> --
> Sframe mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/sframe
>