Re: [AVTCORE] Finishing up framemarking....

["Mo Zanaty (mzanaty)" <[e-mail]>]

Unless anyone objects to the current text in the VP9 draft or Frame Marking draft, then we can consider this issue resolved for both drafts.

Regards,
Mo


From: Jonathan Lennox <[e-mail]>
Date: Monday, July 12, 2021 at 11:18 AM
To: "Mo Zanaty (mzanaty)" <[e-mail]>
Cc: Bernard Aboba <[e-mail]>, "[e-mail]" <[e-mail]>, Justin Uberti <[e-mail]>
Subject: Re: [AVTCORE] Finishing up framemarking....

I agree it’s unfortunate that it’s not directly possible to determine discardability from the RTP payload header, but as you say, a marker can parse the uncompressed VP9 header.  (It’s a fairly straightforward structure to parse, at least as video codecs go.)

Also, I’d imagine that in many cases the frame marking header is being generated by the original generator of the VP9 bitstream; in this case, if it knows the encoding structure in use, it can know what’s discardable that way.

In the worst case, it’s always safe to conservatively mark nothing at all as discardable, of course.

I don’t think doing a substantive change to VP9 would be a good idea.


On Jul 12, 2021, at 12:52 AM, Mo Zanaty (mzanaty) <[e-mail]<mailto:[e-mail]>> wrote:

Thanks, Bernard, for reiterating this VP9 issue so we can resolve it.

Note the latest VP9 draft N bit is unrelated to the VP8 N bit or the Frame Marking D (Discardable) bit. The VP9 N bit signals which other frames this frame refers to, not whether or not any future frames will refer to this frame. It does NOT signal if the current frame is a discardable (non-reference) frame or a non-discardable (reference) frame, as in VP8 and Frame Marking.

VP9 currently requires using the refresh_frame_flags in the VP9 payload uncompressed header to determine if a frame is a discardable non-reference frame (when all bits are zero). The Frame Marking draft describes this. If that is acceptable to all, we can resolve both VP9 and Frame Marking as-is. If VP9 users want an easier way to determine if a frame is discardable without parsing everything before the refresh_frame_flags, that would need a substantive change in the VP9 draft, which would then be referenced in the Frame Marking draft.

Regards,
Mo

From: avt <[e-mail]<mailto:[e-mail]>> on behalf of Bernard Aboba <[e-mail]<mailto:[e-mail]>>
Date: Sunday, July 11, 2021 at 11:17 AM
To: "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>
Cc: Justin Uberti <[e-mail]<mailto:[e-mail]>>, Jonathan Lennox <[e-mail]<mailto:[e-mail]>>
Subject: Re: [AVTCORE] Finishing up framemarking....

An update. The VP9 RTP payload no longer references framemarking, so framemarking is not holding up publication (LRR is the MISREF).

The text that Mo references is now in Section 9 of the VP9 spec (https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section9)


9<https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section-9>.  Congestion Control



   Congestion control for RTP SHALL be used in accordance with RFC 3550<https://datatracker.ietf.org/doc/html/rfc3550>

   [RFC3550<https://datatracker.ietf.org/doc/html/rfc3550>], and with any applicable RTP profile; e.g., RFC 3551<https://datatracker.ietf.org/doc/html/rfc3551>

   [RFC3551<https://datatracker.ietf.org/doc/html/rfc3551>].  The congestion control mechanism can, in a real-time

   encoding scenario, adapt the transmission rate by instructing the

   encoder to encode at a certain target rate.  Media aware network

   elements MAY use the information in the VP9 payload descriptor in

   Section 4.2<https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section-4.2> to identify non-reference frames and discard them in

   order to reduce network congestion.  Note that discarding of non-

   reference frames cannot be done if the stream is encrypted (because

   the non-reference marker is encrypted).





Section 4.2 now includes definition of an N bit:



   Reference indices:  When P and F are both set to one, indicating a

      non-key frame in flexible mode, then at least one reference index

      MUST be specified as below.  Additional reference indices (total

      of up to 3 reference indices are allowed) may be specified using

      the N bit below.  When either P or F is set to zero, then no

      reference index is specified.



      P_DIFF:  The reference index (in 7 bits) specified as the relative

         PID from the current picture.  For example, when P_DIFF=3 on a

         packet containing the picture with PID 112 means that the

         picture refers back to the picture with PID 109.  This

         calculation is done modulo the size of the PID field, i.e.,

         either 7 or 15 bits.  A P_DIFF value of 0 is invalid.



      N:  1 if there is additional P_DIFF following the current P_DIFF.

On Sun, Jul 11, 2021 at 8:04 AM Bernard Aboba <[e-mail]<mailto:[e-mail]>> wrote:
On Thursday, March 11, 2021, Mo posted a question relating to VP9:
https://mailarchive.ietf.org/arch/msg/avt/JA4b0XAllPtrSiTRxr8JIeGWR7E/

I believe this is the only remaining issue on the framemarking document. Since framemarking is holding up publication of the VP9 RTP payload as an RFC, it would be good to move this toward resolution at IETF 111 (or ideally, before).

--------------

VP9 experts,



While updating frame marking to add back VP9, I had difficulty finding an indicator for discardable non-reference frames in the VP9 RTP payload draft to map to the frame marking “D” bit. VP8 has the “N” bit for this in the first payload byte (payload descriptor). But VP9 lacks anything similar, although section 8 thinks this should be in section 4.2 (but there is nothing there or elsewhere).



https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-8

   ...Media aware network elements MAY use the information in the VP9 payload descriptor in Section 4.2<https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2><http://4.2%3Chttps/tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2%3E> to identify non-reference frames and discard them in order to reduce network congestion.  Note that discarding of non-reference frames cannot be done if the stream is encrypted (because the non-reference marker is encrypted).



Is this an oversight? Or was the non-ref indicator intentionally removed? If so, why?



>From my work on AV1, I know how the 8 reference frames work in AV1/VP9, so I devised a solution for frame marking. But it seems like a hack to require this knowledge and go deep into the bitstream uncompressed header (not RTP payload descriptor) to extract this information. Is this really what we must do for VP9? Here is what I put in frame marking version -12, which seems like a hack:



https://tools.ietf.org/html/draft-ietf-avtext-framemarking-12#section-3.3.1

   The D bit MUST be 1 if the refresh_frame_flags in the VP9 payload uncompressed header are all 0, otherwise it MUST be 0.



Reading the refresh_frame_flags from the payload requires some payload parsing and checks since it’s not a fixed offset, and requires knowing subtle nuances like key frames implicitly force all 8 flags to 1. If implementers have been confused by something as simple as TL0PICIDX++, they will likely stumble on this as well.



Is it possible to revive the VP8 N bit? This would simplify frame marking, AV1 DD, GCC, L2/L3 QoS mappings, or anything else that wants to know about discardable data.



Regards,

Mo