Re: [AVTCORE] HEVC in WebRTC: PR #17 (Guidance on RPSI)

[Stephan Wenger <[e-mail]>]

Hi Bernard,
Inline in blue for those with a reasonably modern HTML-capable MUA, indented for the rest.
Allow me to comment here as I do not use github.  Feel free to copy-past the comments over.
Sorry for the word count.
Stephan


From: avt <[e-mail]> on behalf of Bernard Aboba <[e-mail]>
Date: Wednesday, September 27, 2023 at 20:45
To: IETF AVTCore WG <[e-mail]>
Subject: [AVTCORE] HEVC in WebRTC: PR #17 (Guidance on RPSI)

Based on the discussion at the Virtual Interim, I have submitted PR #17:

The proposed text is as follows:

[RFC7798] Section 8.3 specifies the use of the Reference Picture Selection
Indication (RPSI) in H.265. As noted in "Media Transport and Use of RTP in WebRTC"
[RFC8834] Section 5.1.4:

"Receivers that detect that encoder-decoder synchronization has been lost SHOULD
generate an RPSI feedback message if the codec being used supports reference-picture
selection.


Procedurally, based on recent experience, a SHOULD without explanation of that level of mandation (typically, “SHOULD unless blah blah”) will raise eyebrows.
Technically, in this case, a good “unless” could be something like “unless the receiver has knowledge that the transmitter does not support RPSI.  Such knowledge can be established through a) the capability exchange, but also b) through previously sent RPSI requests that were not replied to by the transmitter through the use of a non-IRAP picture.”
To explain, when you have learned that you lost a picture and a fallback to a previous identified picture is desirable, a transmitter capable of using reference picture selection would, of course, use such a reference picture.  However, a slightly less smart transmitter may signal at the cap exchange time that RPSI is supported, only to treat RPSI as another form of Picture Loss Indication (PLI) and respond with an IDR/IRAP.  That’s totally fine and within language and intent of RFC 5104 and 8082.  However, if such a behavior is observed repeatedly, it’s a smart strategy for the receiver to discontinue the use of RPSI and fallback to sending PLI, and internally label that transmitter as RPSI-incapable.  That may come handy in multipoint scenarios.





An RTP packet-stream sender that receives such an RPSI message SHOULD
act on that messages to change the reference picture, if it is possible to do
so within the available bandwidth constraints and with the codec being used."


There’s first the “SHOULD” without an “unless”, which will create heartache.  However, we have a deeper problem here regarding the “bandwidth constraints”.  That’s tricky, as the typical IETFer probably does not understand some of what I write below without reading up on video codec technology, and getting into such details in a profiling document like yours seems questionable.  Still, as written, the guidance provided is misleading.  As a transmitter, you do not have a practical option to move on with business as usual once you have received an RPSI.  You have to react, or the user experience will suffer—badly.  Your language could be read as if doing nothing were an option if you are in a congestion situation.  It’s not.
I assume that language regarding “available bandwidth constraints” is present to appease the congestion control crowd for the usual political reasons, and for that purpose it’s probably fine, though they may also complain about the “SHOULD”.  They will hopefully read it, and nod, and go on with their business.  A real-world implementer will instead rely on network elasticity and ignore congestion control advise and go on with their business also.  Which, so far, hasn’t brought the Internet down, either, despite gazillion of bits being sent every day by real-time video codecs.
I would remove the bandwidth constraint language.  If that’s not possible, I think you need more explanation that your otherwise laudably concise document offers elsewhere.  That explanation could be along the lines “if you receive RPSI, you know something is wrong in the decoder’s reference picture buffer, and you need to react or watch your product losing market share due to bad user experience.  That reaction will cost bits on the wire, and that may upset the congestion control mechanics and violate constraints.  Still, you have to spend those bits, otherwise the user experience will be dismal.  If you choose to stay within the bandwidth budget the congestion control algorithm suggests, or if you are up to a hard limit, you have a few options, though none of them particularly palatable.  An encoder capable of RPSI necessarily is a real-time encoder, which has means to reduce its sending bitrate on a per picture basis.  Two common strategies are to skip source pictures (reducing the frame rate on the wire), or dial up the quantizer (hence reduce user-perceived quality, but not as badly as it would be if an RPSI were ignored).  Those, techniques, in combination with the use a reference picture indicated as useful in the RPSI, are the best known reaction by a transmitter to an RPSI under congestion.  Less good but still acceptable and common implementation practice in some systems is to wait for bitrate budget and then send an IR/IRAP.  The user has to cope with the frozen picture.  Worst case, you can terminate the video.





Review comments welcome:

https://github.com/aboba/hevc-webrtc/pull/17<https://github.com/aboba/hevc-webrtc/pull/9>