Re: [rtcweb] [avtext] Fwd: New Version Notification for draft-fineberg-avtext-temporal-layer-ext-00.txt

Adam Fineberg <fineberg@vline.me> Fri, 19 July 2013 22:54 UTC

Return-Path: <fineberg@vline.me>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6F52011E81CA for <rtcweb@ietfa.amsl.com>; Fri, 19 Jul 2013 15:54:15 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.438
X-Spam-Level:
X-Spam-Status: No, score=-3.438 tagged_above=-999 required=5 tests=[AWL=0.160, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XaJw5Zao7mPN for <rtcweb@ietfa.amsl.com>; Fri, 19 Jul 2013 15:54:11 -0700 (PDT)
Received: from mail-pa0-f42.google.com (mail-pa0-f42.google.com [209.85.220.42]) by ietfa.amsl.com (Postfix) with ESMTP id 4C2D221F8FF3 for <rtcweb@ietf.org>; Fri, 19 Jul 2013 15:53:42 -0700 (PDT)
Received: by mail-pa0-f42.google.com with SMTP id rl6so4948863pac.29 for <rtcweb@ietf.org>; Fri, 19 Jul 2013 15:53:42 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:x-gm-message-state; bh=kNoA5LxJKZ20LHMjfuI3tGyrQXkybzJXu8+etVBNzk8=; b=gAIjVhi/5LH//QPzB2INyflpuLglbwgIu3vHNA1bvsXg9soDyTZ62cuHpJpuPDBNiK Q5pK/qc+Ew5WbPAmM+nEzpIOfA8dhNz/8gM5e2Zk6TvWaCWA2WXYnFlDVdRgGkkRhBPD zx6h06Mre/2end5zCVld8kpyRNRPjKXDpXJAfaZWqbAgyPUqLFXUk0IUgJGuIYabVDOe NwJp9NBcB1Q1Eb/GiBwY8zPBejJSjK6phaIL2cIi2n+iVhauEt3xn+sRFkfFvE51XFNZ x/7W1spnE+OE/MUi4Ar4Yq/o0oKCdu0RbO9v4+NU6tYMNjKkXDb05NtYH1J9HzE8AI9/ QETw==
X-Received: by 10.67.21.229 with SMTP id hn5mr20615088pad.135.1374274421939; Fri, 19 Jul 2013 15:53:41 -0700 (PDT)
Received: from Adams-MacBook-Pro.local ([38.102.150.73]) by mx.google.com with ESMTPSA id xl3sm21677782pbb.17.2013.07.19.15.53.38 for <multiple recipients> (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 19 Jul 2013 15:53:40 -0700 (PDT)
Message-ID: <51E9C371.2090905@vline.me>
Date: Fri, 19 Jul 2013 15:53:37 -0700
From: Adam Fineberg <fineberg@vline.me>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130620 Thunderbird/17.0.7
MIME-Version: 1.0
To: Stephan Wenger <stewe@stewe.org>
References: <CE0F0BEB.9F95D%stewe@stewe.org>
In-Reply-To: <CE0F0BEB.9F95D%stewe@stewe.org>
Content-Type: multipart/alternative; boundary="------------010603030307070406090203"
X-Gm-Message-State: ALoCoQnoMZfNOpxgyk84WWfk9sfg8vx6bbULiGeS857ZOJL8IUj59Z0TcoSkWk1nrPjz7Snkv12m
Cc: "avtext@ietf.org" <avtext@ietf.org>, "rtcweb@ietf.org" <rtcweb@ietf.org>
Subject: Re: [rtcweb] [avtext] Fwd: New Version Notification for draft-fineberg-avtext-temporal-layer-ext-00.txt
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 19 Jul 2013 22:54:15 -0000

Stephan,

On 7/19/13 3:44 PM, Stephan Wenger wrote:
> Hi,
>
> From: Adam Fineberg <fineberg@vline.me <mailto:fineberg@vline.me>>
> Date: Friday, 19 July, 2013 15:12
> To: Stephan Wenger <stewe@stewe.org <mailto:stewe@stewe.org>>
> Cc: Justin Uberti <juberti@google.com <mailto:juberti@google.com>>, 
> Bernard Aboba <bernard_aboba@hotmail.com 
> <mailto:bernard_aboba@hotmail.com>>, "avtext@ietf.org 
> <mailto:avtext@ietf.org>" <avtext@ietf.org <mailto:avtext@ietf.org>>, 
> "rtcweb@ietf.org <mailto:rtcweb@ietf.org>" <rtcweb@ietf.org 
> <mailto:rtcweb@ietf.org>>
> Subject: Re: [avtext] [rtcweb] Fwd: New Version Notification for 
> draft-fineberg-avtext-temporal-layer-ext-00.txt
>
> Stephan,
>
> Thanks for the info and the reference.  I'm not sure I follow as I'm 
> not at all familiar with H.265.  I'll review the reference and see 
> what I can figure.
>
> StW: Good luck :-)
>
> It seems though to me that you are suggesting that except in the 
> simple case, that the data for H.265 would not be well suited to a 
> header extension, am I understanding you correctly?  There is no 
> reason the middlebox couldn't get out of band signaling of the VPS as 
> you mention but that would not be within the scope of this header 
> extension.
>
> StW: well, if you would copy the layer_id into your header extension 
> (just as you need to do for the simple case), a really smart middle 
> box could use this information just as a decoder uses 
> it, assuming that it intercepted the VPS in the first place.  Insofar, 
> I wouldn't rule out the second option on technical grounds.  Whether 
> any of the actual products would bother to do that, ever, is another 
> question.  I think the case ought to be documented, though.  I can 
> help drafting text.

Any help is appreciated.

> While we are at it: doing this right could mean that you need multiple 
> specs.  First, a generic header extension mechanism dedicated to side 
> information required for pruning of RTP packet streams—ideally not 
> only for scalable video, although that is the main customer today. 
>  And second, for each "payload" (at present we are talking about 
> H.264/SVC, H.265v1 (HEVC), H.265v2 (including scalable and 3D 
> extensions, which are not yet finalized), VP8, and VP9 (I know little 
> about the latter), plus Daala, and whatnot) a mapping of the bits 
> available in the generic header extension to the bits in the payload 
> itself (NAL header and VPS in case of H.265, NALU header in SVC, and 
> the fields you mention for VP8).

I was just thinking about that since it seems like the extension payload 
maybe completely specific and therefore not suitable for the same spec.  
The problem in my mind with taking the hierarchical a[[roach you mention 
is that I'm not sure what if any common data is suitable accross all 
codecs and since there is already a spec for the general header 
extension, the generic one you mention might be fairly redundant.

I'm certainly open to your thoughts though.

Regards,
Adam

>
> Stephan
>
>
> Any insights are appreciated.
>
> Regards,
> Adam
>
> On 7/19/13 8:33 AM, Stephan Wenger wrote:
>> Hi,
>> I also believe that 16 bits should be enough.  For H.264 and VP8 that 
>> has already been demonstrated.  For H.265, some initial thoughts 
>> below.  Apologies for the word-count.
>>
>> The scalable version of H265 (called SHVC) is currently under 
>> development.  The current working draft can be found here: 
>> http://phenix.int-evry.fr/jct/doc_end_user/documents/13_Incheon/wg11/JCTVC-M1008-v3.zip. 
>>  Therein, the options for defining layering structures are 
>> considerably more complex.  To start, we have 3 bits for the temporal 
>> ID in the NAL unit header of the H.265 version 1 (HEVC) base 
>> specification (temporal scalability is already nicely supported in 
>> version 1).  Just like in SVC.  In the scalable extension, the NAL 
>> unit header contains a six bit field that points into a data 
>> structure known as "Video Parameter Set" (VPS).  Inside the VPS, 
>> those six bits are mapped to to a position in a directed graph 
>> (specified through "dimension_id[][]"), which tells you about the 
>> reference relationship of the layer in question and its parent layer. 
>>  One can recursively follow the graph to determine what used to be 
>> called dependency_id, quality_id, view_id, and whatnot.  The six bit 
>> pointer field can (or: is to be when possible) organized by the 
>> encoder such that it is prudent for a middle box to throw away NAL 
>> units (belonging to layers) with higher values of the six bit field 
>> first, before throwing away NAL units with lower values.  Relying on 
>> this feature, 3+6 bits == 9 bits should be fine for the header extension.
>>
>> That said, the ordering by the encoder is just a recommendation, and 
>> there may well be cases where different pruning strategies may be 
>> advisable.  For example, a layering structure could be constructed 
>> that expands into two branches, one using 2D scalable tools only, the 
>> other including view_id for multi view coding.  By looking at the six 
>> bit field alone, a middle box will not be able to meaningfully remove 
>> NAL units belonging to one of the branches completely while pruning 
>> the other branch.  In order to meaningfully deal with that scenario, 
>> there would be two options: one to represent the dimension_id[][] 
>> (and associated control info) in the header extension, or require the 
>> middle box to have access to the VPS and be able to interpret its 
>> content.  The further could take considerably more than 16 bits and 
>> we would be talking about a variable length data structure.  The 
>> latter requires the middle box to have state and a mechanism to 
>> intercept the VPS (through signaling—as the encrypted in-band VPS 
>> would not be useful under the assumption that the middle box does not 
>> have the key to the media—which is the motivation of the draft in the 
>> first place).  I personally don't mind at all the second mechanism, 
>> as I'm a big fan of out-of-band parameter set transmission and any 
>> middle box must be in the signaling path anyway to meaningfully 
>> manipulate RTP.  I do not like the first option due to its variable, 
>> and possibly substantial, overhead.
>>
>> Stephan
>>
>>
>> From: Justin Uberti <juberti@google.com <mailto:juberti@google.com>>
>> Date: Friday, 19 July, 2013 06:32
>> To: Bernard Aboba <bernard_aboba@hotmail.com 
>> <mailto:bernard_aboba@hotmail.com>>
>> Cc: "avtext@ietf.org <mailto:avtext@ietf.org>" <avtext@ietf.org 
>> <mailto:avtext@ietf.org>>, "rtcweb@ietf.org <mailto:rtcweb@ietf.org>" 
>> <rtcweb@ietf.org <mailto:rtcweb@ietf.org>>
>> Subject: Re: [rtcweb] Fwd: New Version Notification for 
>> draft-fineberg-avtext-temporal-layer-ext-00.txt
>>
>> Agree those are the right codecs to design for. Since in each case 
>> there are fairly low limits on the number of supported layers (i.e. 3 
>> spatial layers for SVC), I think it should be possible to pack the 
>> temporal, spatial, quality layer ids into 16 bits.
>>
>>
>> On Fri, Jul 19, 2013 at 1:56 AM, Bernard Aboba 
>> <bernard_aboba@hotmail.com <mailto:bernard_aboba@hotmail.com>> wrote:
>>
>>     If we can support VP8/9 as well as H.264/5 SVC
>>     that would be a start. It seems doable to me.
>>
>>     On Jul 18, 2013, at 8:34 PM, "Adam Fineberg" <fineberg@vline.me
>>     <mailto:fineberg@vline.me>> wrote:
>>
>>>     Bernard,
>>>
>>>     Are there other codecs you are thinking should be supported? If
>>>     it's generalized I would think we want to be able to cover all
>>>     known scalable codecs. I'll look into the H264/SVC fields to see
>>>     how to encode them in a generalized header.
>>>
>>>     Regards,
>>>     Adam
>>>
>>>     On 7/18/13 7:40 PM, Bernard Aboba wrote:
>>>>     I think it may be possible to generalize this. For example, for
>>>>     H.264/SVC which can support temporal, spatial and quality
>>>>     scalability, you would need the quality_id and dependency_id in
>>>>     addition to the temporal_id (what you call the temporal layer
>>>>     index).
>>>>
>>>>     ------------------------------------------------------------------------
>>>>     Date: Thu, 18 Jul 2013 08:45:38 -0700
>>>>     From: fineberg@vline.me <mailto:fineberg@vline.me>
>>>>     To: bernard_aboba@hotmail.com <mailto:bernard_aboba@hotmail.com>
>>>>     CC: rtcweb@ietf.org <mailto:rtcweb@ietf.org>; avtext@ietf.org
>>>>     <mailto:avtext@ietf.org>
>>>>     Subject: Re: [rtcweb] Fwd: New Version Notification for
>>>>     draft-fineberg-avtext-temporal-layer-ext-00.txt
>>>>
>>>>     Bernard,
>>>>
>>>>     Good question.  I'm not familiar enough with the parameter
>>>>     requirements of all other scalable codecs to be able to
>>>>     generalize.  If you'd like to help specify them, I'd be fine
>>>>     revising the draft to generalize.
>>>>
>>>>     Regards,
>>>>     Adam
>>>>
>>>>     On 7/17/13 8:26 PM, Bernard Aboba wrote:
>>>>
>>>>         Since the need is not codec specific (e.g. it arises with
>>>>         any codec supporting temporal, spatial and quality
>>>>         scalability), why
>>>>          a VP8-specific RTP extension?
>>>>
>>>>         ------------------------------------------------------------------------
>>>>         Date: Wed, 17 Jul 2013 17:09:46 -0700
>>>>         From: fineberg@vline.me <mailto:fineberg@vline.me>
>>>>         To: rtcweb@ietf.org <mailto:rtcweb@ietf.org>
>>>>         Subject: [rtcweb] Fwd: New Version Notification for
>>>>         draft-fineberg-avtext-temporal-layer-ext-00.txt
>>>>
>>>>         Hi,
>>>>
>>>>         I'm working on WebRTC services and have found that while
>>>>         developing services that forward VP8 video streams if we
>>>>         want to take advantage of the VP8 temporal scaling we must
>>>>         get the temporal layer information from the RTP header
>>>>         which requires us to decrypt the SRTP packets. This is
>>>>         undesirable both because the middle-box needs to have
>>>>         access to the keys as well as the because of the added
>>>>         overhead of the decrypt/encrypt cycle. This draft proposes
>>>>         an RTP header extension that will allow us to use the VP8
>>>>         temporal layer information included in the header extension
>>>>         and therefore do forwarding without SRTP decryption.
>>>>         Comments welcome.
>>>>
>>>>         Regards,
>>>>         Adam Fineberg
>>>>         fineberg at vline.com <mailto:fineberg%20at%20vline.com>
>>>>
>>>>         -------- Original Message --------
>>>>         Subject: 	New Version Notification for
>>>>         draft-fineberg-avtext-temporal-layer-ext-00.txt
>>>>         Date: 	Tue, 09 Jul 2013 10:02:05 -0700
>>>>         From: 	internet-drafts at ietf.org
>>>>         <mailto:internet-drafts%20at%20ietf.org>
>>>>         To: 	Adam Fineberg<fineberg at vline.com>
>>>>         <mailto:fineberg%20at%20vline.com>
>>>>
>>>>
>>>>
>>>>         A new version of I-D, draft-fineberg-avtext-temporal-layer-ext-00.txt
>>>>         has been successfully submitted by Adam Fineberg and posted to the
>>>>         IETF repository.
>>>>
>>>>         Filename:	 draft-fineberg-avtext-temporal-layer-ext
>>>>         Revision:	 00
>>>>         Title:		 A Real-Time Transport Protocol (RTP) Header Extension for VP8 Temporal Layer Information
>>>>         Creation date:	 2013-07-08
>>>>         Group:		 Individual Submission
>>>>         Number of pages: 6
>>>>         URL:http://www.ietf.org/internet-drafts/draft-fineberg-avtext-temporal-layer-ext-00.txt
>>>>         Status:http://datatracker.ietf.org/doc/draft-fineberg-avtext-temporal-layer-ext
>>>>         Htmlized:http://tools.ietf.org/html/draft-fineberg-avtext-temporal-layer-ext-00
>>>>
>>>>
>>>>         Abstract:
>>>>             This document defines a mechanism by which packets of Real-Time
>>>>             Tranport Protocol (RTP) video streams encoded with the VP8 codec can
>>>>             indicate, in an RTP header extension, the temporal layer information
>>>>             about the frame encoded in the RTP packet.  This information can be
>>>>             used in a middlebox performing bandwidth management of streams
>>>>             without requiring it to decrypt the streams.
>>>>
>>>>
>>>>         _______________________________________________ rtcweb
>>>>         mailing list rtcweb@ietf.org <mailto:rtcweb@ietf.org>
>>>>         https://www.ietf.org/mailman/listinfo/rtcweb
>>>>
>>>>
>>>>     -- 
>>>>     Regards,
>>>>     Adam
>>>
>>
>>     _______________________________________________
>>     rtcweb mailing list
>>     rtcweb@ietf.org <mailto:rtcweb@ietf.org>
>>     https://www.ietf.org/mailman/listinfo/rtcweb
>>
>>
>>
>>
>> _______________________________________________
>> avtext mailing list
>> avtext@ietf.orghttps://www.ietf.org/mailman/listinfo/avtext
>
> -- 
> Regards,
> Adam

-- 
Regards,
Adam