Re: [AVTCORE] Some (SFrame RTP encapsulation) questions to ponder

While updating frame marking to add back VP9, I checked the actual specs and discovered that VP9 and VP8 have identical language on TL0PICIDX, and both merely say it “MUST be incremented” but not necessarily by 1. Any implementation that requires +1 seems incorrect unless I’m missing something. There is an additional informative note in VP8: “This will likely require rewriting values of TL0PICIDX and KEYIDX after the splice.” This also does not require +1, and also seems to apply equally to VP9 for TL0PICIDX, and was likely removed from VP9 because it lacks KEYIDX, not because it has any different semantics for TL0PICIDX.

VP9: https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2
      TL0PICIDX:  8 bits temporal layer zero index.  TL0PICIDX is only
         present in the non-flexible mode (F = 0).  This is a running
         index for the temporal base layer pictures, i.e., the pictures
         with TID set to 0.  If TID is larger than 0, TL0PICIDX
         indicates which temporal base layer picture the current picture
         depends on.  TL0PICIDX MUST be incremented when TID is equal to
         0.  The index SHOULD start on a random number, and MUST restart
         at 0 after reaching the maximum number 255.

VP8: https://tools.ietf.org/html/rfc7741#section-4.2
      TL0PICIDX:  8 bits temporal level zero index.  TL0PICIDX is a
         running index for the temporal base layer frames, i.e., the
         frames with TID set to 0.  If TID is larger than 0, TL0PICIDX
         indicates on which base-layer frame the current image depends.
         TL0PICIDX MUST be incremented when TID is 0.  The index MAY
         start at a random value, and it MUST wrap to 0 after reaching
         the maximum number 255.  Use of TL0PICIDX depends on the
         presence of TID.  Therefore, it is RECOMMENDED that the TID be
         used whenever TL0PICIDX is.
…snip…
   Informative note:  Implementations doing splicing of VP8 streams will
      have to make sure the rules for incrementing TL0PICIDX and KEYIDX
      are obeyed across the splice.  This will likely require rewriting
      values of TL0PICIDX and KEYIDX after the splice.

Regards,
Mo

From: avt <avt-bounces@ietf.org> on behalf of Alexandre GOUAILLARD <agouaillard@gmail.com>
Date: Thursday, March 11, 2021 at 3:47 AM
To: Sergio Murillo <sergio.garcia.murillo@gmail.com>
Cc: "avt@ietf.org" <avt@ietf.org>
Subject: Re: [AVTCORE] Some (SFrame RTP encapsulation) questions to ponder

Sergio,

I think you had done this analysis for h264, vp8 and vp9 two years ago. There are some existing  slides about it.

If Apple/Intel add a slide for h265 since they support it in safari, we add av1, Stefan bring in h266, we have more or less most of the codecs of interest and enough data to judge.

We could have three sections:
1 - what do you need to use to go into a generic packetizer and how much is codec specific.
2 - what do You need to duplicate in an rtp header extension for an SFU to work.
I see three different things:
- base for encrypted non-SVC payload (frame marking)
- scalability structure (payload encrypted or not).
a/ I have not had the time to look at Stefan a proposal for the new MPEG codecs but his messages on the list were going in the same direction (higher complexity with full SVC than with temporal scalability only).
b/ I do not know how far appart the AV1 structure and the vvc or ecc would be, if there is a way to make it generic, and if it a even a goal we would want to have.
c/ I understand there was originally a framemarking part to the vvc rtp payload proposal, I m wondering if there was a gap when trying to represent all SVC structures the vvc bistream otherwise provides.
- sfu optimisation structures, eg for av1 the Instant decidability decision structure (payload encrypted or not), whic is not critical for sfu operation but allow much faster adaptation and smarter CC.

If we go down that path that could mean a codec specific version of each of:
- Payload format
- Sframe rtp packetizer
- rtp extension header

With the following triplet already more or less working well together already:
[h264, generic, framemarking]
[VP8, generic, framemarking]
[VP9, generic, DD]
[AV1, generic, DD]

Now in terms of security for the encrypted case, that means the presence or absence of a given rtp header extension leaks info about e.g which codec is being used. I m not sure this is something w can completely avoid anyway.

Sent from my iPhone

On 11 Mar 2021, at 15:15, Sergio Garcia Murillo <sergio.garcia.murillo@gmail.com> wrote:

El jue, 11 mar 2021 a las 7:22, Bernard Aboba (<bernard.aboba@gmail.com<mailto:bernard.aboba@gmail.com>>) escribió:

I ask this because there appear to be codecs that are widely deployed (e.g. VP8) that require "hacks" for SFUs to forward, as noted in the presentation at IETF 109.  In the case of VP8, the "hack" was required because of a deficiency in the VP8 decoder specification (e.g. the requirement that PictureIDs be consecutive, not just monotonically increasing).  This means that the "hacks" described can't be eliminated just by coming up with a better RTP forwarding extension.  You'd actually need to change the VP8 specification and upgrade VP8 decoders.

The PictureID and the TL0PICIDX  are defined in rfc7741 RTP Payload Format for VP8 Video and not on the VP8 spec. So, if using an agnostic packetizer instead of the rfc7741 one, the issue just doesn't exist.

Best regards
Sergio
_______________________________________________
Audio/Video Transport Core Maintenance
avt@ietf.org
https://www.ietf.org/mailman/listinfo/avt