Re: [Sframe] Partial decodability and IDUs

[Sergio Garcia Murillo <[e-mail]>]

On 20/11/2020 17:01, Magnus Westerlund wrote:
> Hi,
>
> If one look at this from the RTP payload level what becomes of potential use and
> in some case necessary I think are the following.
>
> The sender will put one or more IDU into independent SFRAMEs and provide the
> meta data to it that can include the following:
>   - Media Stream + Encoder instance
>   - Scalability layer and depdendency information
>   - Decoding order
>   - Associated Capture/Presentation time
>   - timestamp for any consumption related time line (if not redundant)
>   - If it represent a switch point (IDR, Intra etc)
>   - Will produce output (Video marker bit type, i.e. this IDU(s) will result in
> decoded output) (likely for which layers)
>   - Reliability level (How necessary the data is, parameter sets is necessary for
> any decoding, versus a non-referenced slice).
>
> The one or more IDUs is a trade-off if between overhead and what benefits there
> are to encoded different IDUs in indepdendent SFRAMES. For example encoding
> H.26x parameterset NALUs independently depend on how one like to use them, or if
> they should be teamed with their related "slice" NALUS, in a STAP style
> aggregation of NALUs inside the SFRAME.
>
> I think the point is to look at this what would be needed to enable the
> functionality in the RTP payload format and for the SFUs to be able to correctly
> select what SFRAMEs to forward to a particular receiver.

 From my point of view the that the partial decodability is not 
something that the SFU cares about.

The only RTCP recovery I am aware of that could apply for IDUs is the 
slice loss indication which requires macroblock information, which is a 
way lower information layer than the one SFUs works with (it may not 
even be available if payload is encrypted).

Moreover, the SFU will not rely the SLIs received from the receiving 
endpoint as it has no way of tracking SLIs requests from endpoints and 
map them to already sent ones in order to avoiding overflowing the 
encoder with SLIs requests. So I am afraid that the theoretical benefit 
of partial decoding would be reduced a lot in the typical usage of 
SFrame (i.e. no p2p use case).

Also, before adding support from it, I would like to get real 
implementation feedback as it is currently not implemented in 
(lib)webrtc and I have not heard anynone asking for it ever.


> If some implementation makes is easy for themselves and aggregate more in one
> SFRAME that should be possible. It is a trade-off to a large degree between
> functionality, cost of doing repair of missing parts, and complexity to
> implement.


And also complexity to specify.


> There are some basic level which should be very easy to accomplish and never
> should be gone below becauase then one is truly throwing away the basic
> functionalities of RTP.
>
> I still think one needs to write a bit of analysis of the different media
> encoders functionality and what are shared concepts, what have special quirks
> that needs to be thought about. We also I think need to find the terminolgy for
> what the different aspects like I listed above really should be called and how
> they are mapped to different encoders, in audio, video, tactile, etc.
>
> I will note that we didn't make much headway in the RTP streams debate before we
> actually created the taxonomy of that. People where talking past each others.


Are you proposing to do something like rfc7656 but for video codecs? I 
think that would be great, but I also understand that it would be an 
different draft than the packetization or description header, right?


> I will also note that a very interesting aspect of the reliability puzzle for
> these codecs is what you can determine about a missing RTP packet based on that
> you know which SSRC and sequence number that is missing. So if you have a
> scalable codec and transmit all the packets on a single SSRC, then a missing
> sequence number (detected through the gap) will not tell the receiver if it
> needs the packet urgently or not (such a difference between a base layer or the
> highest enchancement layer). In some applications that might not matter in
> otheres it may. So in the later cases you might want to distribute the layers on
> different SSRCs so that one can immediately determine how important the
> information is. I think that possibility will exist if this RTP payload format
> is done correctly.


While I think this information is very valuable (in fact, it is a must 
have from the SFU perspective), I think it belongs to the SFU 
information header extension part and not the packetization part 
(although they could be implemented together).


Best regards

Sergio