Re: [AVTCORE] Comments on “RTP Payload Format for Versatile Video Coding (VVC)”

["Hannuksela, Miska (Nokia - FI/Tampere)" <[e-mail]>]

Excellent. All the technical issues I raised a few weeks ago have now been concluded. Thanks, Stephan, Yago and Ye-Kui, for the constructive discussion. I look forward to a new I-D version and I am happy to help in preparing and reviewing it.

BR,
Miska

From: Stephan Wenger <[e-mail]>
Sent: Tuesday, September 21, 2021 9:49 PM
To: Hannuksela, Miska (Nokia - FI/Tampere) <[e-mail]>om>; IETF AVTCore WG <[e-mail]>
Subject: Re: [AVTCORE] Comments on “RTP Payload Format for Versatile Video Coding (VVC)”

OK.  Option 1 is it.  I count on you to help with the text :-)
S.



From: Miska Hannuksela <[e-mail]<mailto:[e-mail]>>
Date: Tuesday, September 21, 2021 at 11:14
To: Stephan Wenger <[e-mail]<mailto:[e-mail]>>, IETF AVTCore WG <[e-mail]<mailto:[e-mail]>>
Subject: RE: [AVTCORE] Comments on “RTP Payload Format for Versatile Video Coding (VVC)”

Hi Stephan,

Thanks for discussing item 10 and suggesting some options. As far as I can see from your email, you understood my comments at least partly correctly.

Option 2 would be burdensome for receivers  that operate with standard-conforming VVC decoders, since the receivers would need to verify the correctness of the received bitstream and potentially modify it before passing it to the decoder.

If I understood option 3 correctly, a sentence would be added somewhere that the transmitted stream shall comply with the VVC standard. Right?
However, option 3 would leave sprop-sub-layer-id and recv-sub-layer-id quite vague. For example, a receiver could not be sure that by selecting a certain recv-sub-layer-id to the answer, it receives a bitstream that that has the level signaled in the VPS for the sublayer representation of the indicated recv-sub-layer-id.

Thus, I continue to have quite strong preference on option 1.

BR,
Miska

From: Stephan Wenger <[e-mail]<mailto:[e-mail]>>
Sent: Tuesday, September 14, 2021 7:20 PM
To: Hannuksela, Miska (Nokia - FI/Tampere) <[e-mail]<mailto:[e-mail]>>; IETF AVTCore WG <[e-mail]<mailto:[e-mail]>>
Subject: Re: [AVTCORE] Comments on “RTP Payload Format for Versatile Video Coding (VVC)”

Hi Miska, AVT,
Yago kindly reminded me that one of your comments has not yet been addressed, so I try to do so.  Pruning the rest of your original email.

From: avt <[e-mail]<mailto:[e-mail]>> on behalf of Miska Hannuksela <[e-mail]<mailto:[e-mail]>>
Date: Tuesday, August 24, 2021 at 07:41
To: IETF AVTCore WG <[e-mail]<mailto:[e-mail]>>
Subject: [AVTCORE] Comments on “RTP Payload Format for Versatile Video Coding (VVC)”

[…]

10) Section 7.2.2.2, on sprop-sub-layer-id indicating the property of the highest layer only


   o  The offerer MAY include sprop-sub-layer-id which, in case of

      scalable VVC, is interpreted as the highest sub-layer of the

      highest enhancement layer in the OLS indicated by sprop-ols-id.

      The answerer MAY include recv-sub-layer-id which can be used to

      downgrade the sublayer of the highest enhancement layer.  This

      specification does not support downgrading the sub-layer of any

      layers in the OLS that are not the highest layer.

         Informative note: in other words, using this mechanism, an

         answerer can downgrade only the frame rate for the highest

         spatial/quality layer (typically corresponding to the highest

         resolution or bitrate, hence the most complex to decode), but

         not for lower spatial/quality layers.  The answerer must

         support all sublayers for lower layers in the OLS, or reject

         the offer.  That's not a big burden, as the receiver/decoder

         has the option to discard any sublayers it cannot decode,

         irrespective of what is being signalled through offer/answer.


·         This design choice is not obvious. Note that pruning temporal sublayers from the highest layer but not from the reference layer(s) could have undesirable impacts, such as:

o    The decoder outputs reference-layer pictures with TemporalId greater than sprop-sub-layer-id. In SNR/spatial scalability these pictures have lower quality/resolution than the pictures of the highest layer and hence cause temporally fluctuating quality/resolution.

o    If sprop-sub-layer-id is used to input the Htid variable to the decoder as specified in Section 8.1.1 of VVC, the receiver would need to apply the sub-bitstream extraction process of VVC, C.6 to obtain a conforming bitstream.



·         I would think that the maximum of sprop-sub-layer-id (when present) and recv-sub-layer-id (when present) MUST indicate the highest temporal sublayer to be decoded for the bitstream from the offerer to the answerer and MAY be provided as the value of Htid to the VVC decoding process as specified in Section 8.1.1 of VVC. Changes would be needed in Section 7.2.2.2 as well as in sprop-sub-layer-id and recv-sub-layer-id in Section 7.1.

[StW]: I think the main point you raise is that, as currently specified, the O/A process could arrive at an SDP blob that, if read outside the context of the H.266 spec, would allow the sending of a scalable bitstream that’s illegal.  However, I will note that a media description in SDP blobs related to video coding are generally meant as the maximum bitstream complexity.  A sender can always send fewer bits, a lower level, fewer fps, fewer layers, etc. etc., as dictated by its own implementation restrictions, computational constraints, congestion control, and whatnot.  I think bitstream forming constraints of H.266 could conceivably be included in this laundry list.  Then again, your main complaint seems to be that the accepted SDP blob may intuitively describe a scalability ops point that is beyond what an H.266-compliant bitstream could support.
I think there are three solutions to this problem.  In the order of spec impact:

  1.  We could do what you suggest in your second bullet (and thanks for the drop-in text).  Doing so would normatively disallow the O/A process to arrive at signalling sublayer information that could be viewed as illegal when not keeping above thinking process in mind.  The disadvantage is that the O/A implementer would have to dive quite deeply in obscure areas of the H.266 spec.  Those without deep understanding and/or time to make that deep-dive would likely get things wrong.  Then again, anyone implementing scalability probably needs to understand this stuff anyway, and if they get things wrong, we would probably not be worse off than when using one of the alternate solutions below.
  2.  We could add your text to the informative note to the extent that an implementer needs to keep into account bitstream restrictions of section 8.1.1 of the H.266 spec, perhaps with a bit of explanatory language around it.
  3.  We could do nothing, relying on the general observation I made above, namely that what a sender can send in terms of media must be at the intersection of compliance with both the result of the O/A part of this spec, and the H.266 spec.
I’m open to all three option, with a slight preference to #2.

@Miska: did I get this right?
@AVT: anyone else has a preference?

Thanks,
Stephan