Re: [AVTCORE] Finishing up framemarking....

Jonathan Lennox <jonathan.lennox@8x8.com> Mon, 12 July 2021 15:18 UTC

Return-Path: <jonathan.lennox@8x8.com>
X-Original-To: avt@ietfa.amsl.com
Delivered-To: avt@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 495363A0DA1 for <avt@ietfa.amsl.com>; Mon, 12 Jul 2021 08:18:02 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.098
X-Spam-Level:
X-Spam-Status: No, score=-0.098 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, HTTP_ESCAPED_HOST=0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=8x8.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Gfu8bRGI9iFQ for <avt@ietfa.amsl.com>; Mon, 12 Jul 2021 08:17:56 -0700 (PDT)
Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C7DB43A0D99 for <avt@ietf.org>; Mon, 12 Jul 2021 08:17:55 -0700 (PDT)
Received: by mail-qk1-x736.google.com with SMTP id 201so4182157qkj.13 for <avt@ietf.org>; Mon, 12 Jul 2021 08:17:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=8x8.com; s=googlemail; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=st6jo5Aq8TVM6Fcl1HZZMoPuaoSVAbeNiQzVUvE8JsA=; b=HmQAjr7sfZ3bTPPm5C/mvqlyYcUgN1SkKgY104oOjNRucR53qQEeKFSl8jnmHupV4R ldfMHvcBT5nVAnk3knSoekNRVUWdjLEPUw190BkHLp17BEUk/ziUalssWJnurlDLPQOY 1SBTSTs5QS38I8Y+WmTyEIWTukdbNuFUjt4LY=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=st6jo5Aq8TVM6Fcl1HZZMoPuaoSVAbeNiQzVUvE8JsA=; b=QL5eNZmaPwYfDNpvenGfmtttFljZqvHlpr9PXPX78JEx9fpXwz/Tjv8tbnOwr++Ebd POw1bPPDLrQrbQwhacnCnQ67KRebfIoCpB+tcxOv8LdHZ/KpGd6xsCBcf5njP72hMCqv 6x9+5zhA4uk4aLUb5b8sDADTLzL8p2D1ph374VlWm9z2GacOIfFN/iiNX/1CX8xTCc8W pxjNl7d8O/N6HnOt7amSKsg8qT7OCDImT+nukQCsybLJnLducaZCuXubmuMdXmTijZzd KSj/EKHbVPA/hK5+lztC37xP6XuLnyjLs2B+smmEPaHN5GTBb9vujMoU8UZ94WJFVeO+ aQjw==
X-Gm-Message-State: AOAM531wZymoKX1xlGsc+3CZKydq1lOjNXRBFPlGoDORqRWS+B01KK2I zSpG5B2gtVr3V2UXvW2xvWJTkQ==
X-Google-Smtp-Source: ABdhPJwo/yHmeiVdHSfUO3khA8cVz+gikLts/CCBoGjPMjnVCcysppihsSkdMFcBGVXN+o064Sg/yg==
X-Received: by 2002:a05:620a:4490:: with SMTP id x16mr32135941qkp.153.1626103074200; Mon, 12 Jul 2021 08:17:54 -0700 (PDT)
Received: from smtpclient.apple (c-24-0-149-15.hsd1.nj.comcast.net. [24.0.149.15]) by smtp.gmail.com with ESMTPSA id i9sm5332293qtp.50.2021.07.12.08.17.53 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Jul 2021 08:17:53 -0700 (PDT)
From: Jonathan Lennox <jonathan.lennox@8x8.com>
Message-Id: <192A935C-B243-49D6-BD74-8C2C428EC7C3@8x8.com>
Content-Type: multipart/alternative; boundary="Apple-Mail=_4F028A06-CF79-4975-A73C-4B4AFCA64698"
Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.80.0.2.43\))
Date: Mon, 12 Jul 2021 11:17:52 -0400
In-Reply-To: <ABFEF187-9BDF-4FA4-B6A9-D56010B23637@cisco.com>
Cc: Bernard Aboba <bernard.aboba@gmail.com>, IETF AVTCore WG <avt@ietf.org>, Justin Uberti <juberti@gmail.com>
To: "Mo Zanaty (mzanaty)" <mzanaty@cisco.com>
References: <CAOW+2dvLjy-XyCP6=nKKYM-Kveb=RLjEWNVVYKtaoje5bo_fmQ@mail.gmail.com> <CAOW+2dvAWdGFC0DwtiKpVoe+9HZckCZuGy=BZsRuhrfHrvhshg@mail.gmail.com> <ABFEF187-9BDF-4FA4-B6A9-D56010B23637@cisco.com>
X-Mailer: Apple Mail (2.3654.80.0.2.43)
Archived-At: <https://mailarchive.ietf.org/arch/msg/avt/MRaPrtbP0FSEXAt53YYm7yWTpe0>
Subject: Re: [AVTCORE] Finishing up framemarking....
X-BeenThere: avt@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Audio/Video Transport Core Maintenance <avt.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/avt>, <mailto:avt-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/avt/>
List-Post: <mailto:avt@ietf.org>
List-Help: <mailto:avt-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/avt>, <mailto:avt-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 12 Jul 2021 15:18:02 -0000

I agree it’s unfortunate that it’s not directly possible to determine discardability from the RTP payload header, but as you say, a marker can parse the uncompressed VP9 header.  (It’s a fairly straightforward structure to parse, at least as video codecs go.)

Also, I’d imagine that in many cases the frame marking header is being generated by the original generator of the VP9 bitstream; in this case, if it knows the encoding structure in use, it can know what’s discardable that way.

In the worst case, it’s always safe to conservatively mark nothing at all as discardable, of course.

I don’t think doing a substantive change to VP9 would be a good idea.

> On Jul 12, 2021, at 12:52 AM, Mo Zanaty (mzanaty) <mzanaty@cisco.com> wrote:
> 
> Thanks, Bernard, for reiterating this VP9 issue so we can resolve it.
>  
> Note the latest VP9 draft N bit is unrelated to the VP8 N bit or the Frame Marking D (Discardable) bit. The VP9 N bit signals which other frames this frame refers to, not whether or not any future frames will refer to this frame. It does NOT signal if the current frame is a discardable (non-reference) frame or a non-discardable (reference) frame, as in VP8 and Frame Marking.
>  
> VP9 currently requires using the refresh_frame_flags in the VP9 payload uncompressed header to determine if a frame is a discardable non-reference frame (when all bits are zero). The Frame Marking draft describes this. If that is acceptable to all, we can resolve both VP9 and Frame Marking as-is. If VP9 users want an easier way to determine if a frame is discardable without parsing everything before the refresh_frame_flags, that would need a substantive change in the VP9 draft, which would then be referenced in the Frame Marking draft.
>  
> Regards,
> Mo
>  
> From: avt <avt-bounces@ietf.org <mailto:avt-bounces@ietf.org>> on behalf of Bernard Aboba <bernard.aboba@gmail.com <mailto:bernard.aboba@gmail.com>>
> Date: Sunday, July 11, 2021 at 11:17 AM
> To: "avt@ietf.org <mailto:avt@ietf.org>" <avt@ietf.org <mailto:avt@ietf.org>>
> Cc: Justin Uberti <juberti@gmail.com <mailto:juberti@gmail.com>>, Jonathan Lennox <jonathan.lennox@8x8.com <mailto:jonathan.lennox@8x8.com>>
> Subject: Re: [AVTCORE] Finishing up framemarking....
>  
> An update. The VP9 RTP payload no longer references framemarking, so framemarking is not holding up publication (LRR is the MISREF).
>  
> The text that Mo references is now in Section 9 of the VP9 spec (https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section9 <https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section9>)
>  
> 9 <https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section-9>.  Congestion Control
>  
>    Congestion control for RTP SHALL be used in accordance with RFC 3550 <https://datatracker.ietf.org/doc/html/rfc3550>
>    [RFC3550 <https://datatracker.ietf.org/doc/html/rfc3550>], and with any applicable RTP profile; e.g., RFC 3551 <https://datatracker.ietf.org/doc/html/rfc3551>
>    [RFC3551 <https://datatracker.ietf.org/doc/html/rfc3551>].  The congestion control mechanism can, in a real-time
>    encoding scenario, adapt the transmission rate by instructing the
>    encoder to encode at a certain target rate.  Media aware network
>    elements MAY use the information in the VP9 payload descriptor in
>    Section 4.2 <https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section-4.2> to identify non-reference frames and discard them in
>    order to reduce network congestion.  Note that discarding of non-
>    reference frames cannot be done if the stream is encrypted (because
>    the non-reference marker is encrypted).
>  
>  
> Section 4.2 now includes definition of an N bit: 
>  
>    Reference indices:  When P and F are both set to one, indicating a
>       non-key frame in flexible mode, then at least one reference index
>       MUST be specified as below.  Additional reference indices (total
>       of up to 3 reference indices are allowed) may be specified using
>       the N bit below.  When either P or F is set to zero, then no
>       reference index is specified.
>  
>       P_DIFF:  The reference index (in 7 bits) specified as the relative
>          PID from the current picture.  For example, when P_DIFF=3 on a
>          packet containing the picture with PID 112 means that the
>          picture refers back to the picture with PID 109.  This
>          calculation is done modulo the size of the PID field, i.e.,
>          either 7 or 15 bits.  A P_DIFF value of 0 is invalid.
>  
>       N:  1 if there is additional P_DIFF following the current P_DIFF.
>  
> On Sun, Jul 11, 2021 at 8:04 AM Bernard Aboba <bernard.aboba@gmail.com <mailto:bernard.aboba@gmail.com>> wrote:
>> On Thursday, March 11, 2021, Mo posted a question relating to VP9: 
>> https://mailarchive.ietf.org/arch/msg/avt/JA4b0XAllPtrSiTRxr8JIeGWR7E/ <https://mailarchive.ietf.org/arch/msg/avt/JA4b0XAllPtrSiTRxr8JIeGWR7E/>
>>  
>> I believe this is the only remaining issue on the framemarking document. Since framemarking is holding up publication of the VP9 RTP payload as an RFC, it would be good to move this toward resolution at IETF 111 (or ideally, before). 
>>  
>> --------------
>> VP9 experts,
>>  
>> While updating frame marking to add back VP9, I had difficulty finding an indicator for discardable non-reference frames in the VP9 RTP payload draft to map to the frame marking “D” bit. VP8 has the “N” bit for this in the first payload byte (payload descriptor). But VP9 lacks anything similar, although section 8 thinks this should be in section 4.2 (but there is nothing there or elsewhere).
>>  
>> https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-8 <https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-8>
>>    ...Media aware network elements MAY use the information in the VP9 payload descriptor in Section 4.2<https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2> <http://4.2%3Chttps/tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2%3E> to identify non-reference frames and discard them in order to reduce network congestion.  Note that discarding of non-reference frames cannot be done if the stream is encrypted (because the non-reference marker is encrypted).
>>  
>> Is this an oversight? Or was the non-ref indicator intentionally removed? If so, why?
>>  
>> >From my work on AV1, I know how the 8 reference frames work in AV1/VP9, so I devised a solution for frame marking. But it seems like a hack to require this knowledge and go deep into the bitstream uncompressed header (not RTP payload descriptor) to extract this information. Is this really what we must do for VP9? Here is what I put in frame marking version -12, which seems like a hack:
>>  
>> https://tools.ietf.org/html/draft-ietf-avtext-framemarking-12#section-3.3.1 <https://tools.ietf.org/html/draft-ietf-avtext-framemarking-12#section-3.3.1>
>>    The D bit MUST be 1 if the refresh_frame_flags in the VP9 payload uncompressed header are all 0, otherwise it MUST be 0.
>>  
>> Reading the refresh_frame_flags from the payload requires some payload parsing and checks since it’s not a fixed offset, and requires knowing subtle nuances like key frames implicitly force all 8 flags to 1. If implementers have been confused by something as simple as TL0PICIDX++, they will likely stumble on this as well.
>>  
>> Is it possible to revive the VP8 N bit? This would simplify frame marking, AV1 DD, GCC, L2/L3 QoS mappings, or anything else that wants to know about discardable data.
>>  
>> Regards,
>> Mo