Re: [rtcweb] [payload] VP8 payload, decoder processing capabilities (was Re: Resolution negotiation - a contribution)

[Harald Alvestrand <[e-mail]>]

On 03/25/2013 03:53 PM, Cullen Jennings (fluffy) wrote:
> I'd would find it very helpful to see the parameterization of Google hardware implementations. It been said that anyone can get the CHDL/Verilog for this but I can't get it. Looking at that would help understand what are limitations of at least one hardware implementation. I think that would shed a bunch of light on what might be needed.

Cullen, I still haven't seen the note from you with a copy of the note 
from Aki that said what the issue is with you getting access to the 
hardware design stuff.

The message you quote below is almost one year old, and we added two new 
parameters to the VP8 SDP declaration after that specifically to address 
Stefan's point, and included the following text:

6.2.2.  Offer/Answer Considerations

    The VP8 codec offers a decode complexity that is roughly linear with
    the number of pixels encoded.  In some practical applications, there
    will be a need for negotiating frame rate and resolution, provided by
    the OPTIONAL parameters "max-fs" and "max-fr", in addition to these
    parameters, many practical applications will need a mean to
    communicate the max bitrate.  The SDP endpoints MAY negotiate a
    method to communicate the maximum media bitrate, such as TMMBR in
    [RFC5104], therefore VP8 does not add any new mechanisms for this
    negotiation.  The parameter "max-fr" and "max-fs" are defined in
    Section 6.1, where the macroblock size is 16x16 pixels as defined in
    [RFC6386].  In many practical applications, the max frame size and
    max frame rate are known from other information; if they are not
    constrained by other means, the max-fs and max-fr parameters MUST be
    used to establish these limits.



>
> On Apr 16, 2012, at 14:40 , Stephan Wenger wrote:
>
>> Hi all,
>>
>> For context: Harald and myself have been at odds for a while now about the
>> lack of support for a code point in the VP8 payload that can be used to
>> negotiate a maximum decoder/bitstream complexity.  Specifically, Harald
>> (and other VP8 payload folks) suggested that generic mechanisms, such as
>> the a=framerate attribute of RFC4566 in conjunction with the picture size
>> aspect of the imageattr of RFC 6236 can be used, at least in the rtcweb
>> context.  However, as far as I understood our argument, these two
>> mechanisms in combination are not meant as a limit for decoder complexity
>> (in terms of samples/sec processing rate), but rather as an indication,
>> from receiver to sender, of an upper bound of what is "useful to send".
>> See the email below.  To me, it's quite obvious that an indication of
>> "useful to send" includes "my decoder can handle this"; however, it could
>> be more restrictive in that factors other than decoder horsepower could
>> also be at play, such as screen size, user interface settings, and so on.
>>
>> I believe that the combination of what can be signaled using the above
>> mechanisms should be sufficient for rtcweb.  However, I also believe that
>> it is insufficient for general purpose use, mostly because it requires the
>> support of RFC 6236, which is not exactly a widely deployed technology.
>> Further, the a=framerate attribute is not a particularly useful attribute
>> these days anymore, because variable frame rates, at least for software
>> encoding/decoding, are the norm.
>>
>> In previous posts on the payload list (in response to the VP8 payload
>> WGLC), I have commented on the practical shortcomings of the (lack of)
>> complexity negotiation, and suggested that this needs to be fixed.
>>
>> Two options:
>>
>> 1) codify Harald's mechanism (based on a=framerate and imageattr in the
>> VP8 payload draft, at MUST strength.  "In a declarative context, a
>> prospective media sender supporting this (VP8 payload) specification MUST
>> support RFC 4566 a=framerate and RFC6236 imageattr, and MUST include code
>> points according to both mechanisms to identify the properties of the
>> media stream.  In an offer-answer context, both offerer and answerer
>> receiver supporting this VP8 payload specification MUST support
>> a=framertate and imageattr, and MUST include both in their respective
>> offer/answer messages, so to identify an operation point that will not
>> overload the media decoder's capabilities.
>>
>> The issue with this approach, IMO, is that we are dealing here with three
>> individual code points (framerate, horizontal and vertical picture size),
>> where a single code point ought to be sufficient for determining whether a
>> décor is capable of decoding a stream, at least from a complexity
>> viewpoint).
>>
>> 2) include, in the V8 payload, a negotiable SDP code point indicating the
>> complexity of a stream, in units of samples per second processing
>> requirements or a derivative thereof (such as: levels as used in the MPEG
>> world).  For example, the VP8 payload could include a single, optional,
>> negotiable parameter "SamplePerSecond".  If SamplePerSecond were absent in
>> the SDP, a value of xxxxx must be inferred.  (a sensible value for xxxxx
>> could be, for example 9216000, which is the number of samples per second
>> for VGA resolution at 30 Hz).  If SamplePerSecond is present in a
>> declarative context, it indicates the minimum processing requirements a
>> decoder must support in order to successfully decode the stream.  In a
>> symmetric offer-answer context, SamplePerSecond can be used to "dial down"
>> the complexity of the stream to a value that both encoder and decoder can
>> support.
>>
>> My preference is obviously the second proposal, but I'm willing to help
>> fleshing out either or both of them, just not today :-)
>>
>> Regards,
>> Stephan
>>
>>
>>
>> On 4.13.2012 00:45 , "Harald Alvestrand" <[e-mail]> wrote:
>>
>>> On 04/12/2012 11:13 PM, Stephan Wenger wrote:
>>>> On 4.12.2012 12:08 , "Harald Alvestrand"<[e-mail]>  wrote:
>>>>
>>>>> On 04/12/2012 08:19 PM, Stephan Wenger wrote:
>>>>>> Hi Harald,
>>>>>> Thanks for this strawman.  I believe it should work, but I fail to see
>>>>>> how
>>>>>> a two dimensional negotiation requirement (negotiating max values for
>>>>>> framerate and image size--which, in turn, also has two-dimensional
>>>>>> properties) leads to better interop than a one dimensional negotiation
>>>>>> (pixels per second processing requirement).
>>>>> Stephan,
>>>>>
>>>>> I do not see this (or the requirement from the use-cases document)
>>>>> first
>>>>> and foremost a decoder complexity negotiation; it is a negotiation of
>>>>> how much data it is useful to send, given the recipient's intended use
>>>>> of that data.
>>>> Then such a negotiation should be executed in addition.  Decoder cycle
>>>> requirement do matter in practical implementations.
>>> Feel free to propose language that captures this requirement. As noted,
>>> my I-D fragment doesn't.
>>>
>>>
>>
>> _______________________________________________
>> payload mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/payload
> _______________________________________________
> rtcweb mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/rtcweb