Re: [AVTCORE] Call for Framemarking implementation experience

[Bernard Aboba <[e-mail]>]

On Nov 23, 2020, at 13:49, Jonathan Lennox <[e-mail]> wrote:
> 
> (Responding as an individual.)
> 
> I implemented frame marking at my previous company (Vidyo), specifically for H.264/AVC temporal scalability.
> 
> Frame marking nicely retrofits H.264/AVC to support temporal scalability, in fact better than the H.264/SVC format does (because the H.264/SVC format doesn’t allow you to specify temporal layers on packets containing fragmentation units), and in much more backward-compatible way.  This is why we implemented it at Vidyo, and contributed it to the WebRTC.org codebase.

[BA] How many temporal layers did you support? I take it that you had no problem determining upswitch points for all the supported temporal modes?

> For the reasons that Stephan discussed, however, I never felt that frame marking was particularly suitable for other codecs, where in-payload-header and even in-payload bits need to be inspected and in some cases rewritten. Additionally, frame marking isn’t semantically rich enough (on its own) to express all the information that a fully-featured SFU needs to know about spatially-scalable video, even before newer encoding structures such as K-SVC come into play.
> 
> It may be that the AV1 Descriptor Format will be rich enough, though even there I worry about features features of future codecs that aren’t envisioned by AV1.  A notable example would be (as discussed at last week’s meeting) H.266’s gradual decoder refresh (GDR), with its ambiguity of whether it should be considered as satisfying a keyframe request.

[BA] Jonathan said:
“ where in-payload-header and even in-payload bits need to be inspected and in some cases rewritten”

This sounds like an issue that would be encountered by *any* RTP header extension (e.g.  framemarking, DD, DDv2, etc.). The info could be placed in an RTP header extension so as to allow the SFU to inspect and rewrite it, but then the RTP de-packetizer would need to modify the bitstream before sending it to the decoder to reflect the SFU modified values.  It’s a bit disconcerting to give an untrusted SFU the rights to modify a “protected” bitstream after reception.