Re: [Sframe] Partial decodability and IDUs

[Magnus Westerlund <[e-mail]>]

Hi,

If one look at this from the RTP payload level what becomes of potential use and
in some case necessary I think are the following.

The sender will put one or more IDU into independent SFRAMEs and provide the
meta data to it that can include the following:
 - Media Stream + Encoder instance
 - Scalability layer and depdendency information
 - Decoding order
 - Associated Capture/Presentation time
 - timestamp for any consumption related time line (if not redundant)
 - If it represent a switch point (IDR, Intra etc)
 - Will produce output (Video marker bit type, i.e. this IDU(s) will result in
decoded output) (likely for which layers)
 - Reliability level (How necessary the data is, parameter sets is necessary for
any decoding, versus a non-referenced slice). 

The one or more IDUs is a trade-off if between overhead and what benefits there
are to encoded different IDUs in indepdendent SFRAMES. For example encoding
H.26x parameterset NALUs independently depend on how one like to use them, or if
they should be teamed with their related "slice" NALUS, in a STAP style
aggregation of NALUs inside the SFRAME. 

I think the point is to look at this what would be needed to enable the
functionality in the RTP payload format and for the SFUs to be able to correctly
select what SFRAMEs to forward to a particular receiver. 

If some implementation makes is easy for themselves and aggregate more in one
SFRAME that should be possible. It is a trade-off to a large degree between
functionality, cost of doing repair of missing parts, and complexity to
implement. 

There are some basic level which should be very easy to accomplish and never
should be gone below becauase then one is truly throwing away the basic
functionalities of RTP. 

I still think one needs to write a bit of analysis of the different media
encoders functionality and what are shared concepts, what have special quirks
that needs to be thought about. We also I think need to find the terminolgy for
what the different aspects like I listed above really should be called and how
they are mapped to different encoders, in audio, video, tactile, etc. 

I will note that we didn't make much headway in the RTP streams debate before we
actually created the taxonomy of that. People where talking past each others. 

I will also note that a very interesting aspect of the reliability puzzle for
these codecs is what you can determine about a missing RTP packet based on that
you know which SSRC and sequence number that is missing. So if you have a
scalable codec and transmit all the packets on a single SSRC, then a missing
sequence number (detected through the gap) will not tell the receiver if it
needs the packet urgently or not (such a difference between a base layer or the
highest enchancement layer). In some applications that might not matter in
otheres it may. So in the later cases you might want to distribute the layers on
different SSRCs so that one can immediately determine how important the
information is. I think that possibility will exist if this RTP payload format
is done correctly.

Cheers

Magnus Westerlund