Re: [payload] VP8 payload, decoder processing capabilities (was Re: [rtcweb] Resolution negotiation - a contribution)

["Roni Even" <[e-mail]>]

Hi,
I would like to explain why we have the maxmbps, maxfs, and all these
parameters for H.264 at least my view.

ITU-T H.264 tables A.1 and A.6 define the level values and as such they also
define the maximum resolution allowed for each level and the maximum decoded
picture buffer size and the maximum video bit rate. Now when an endpoint
claims to support a level it must support all parameters. Now according to
table A.6 if you want to support 1080HD video you must support level 4 which
mean you also support video bit rate of 20Mbit/s and 30 frames per second.
Using only the level parameter you cannot say that you support a lower level
but have larger decoded picture buffer size that will allow you to support
1080HD video at 15fps for example. 
As for the VP8 case note also the need to know the maximum frame size which
is important for the required decoded picture buffer size. This may look not
important for low resolution but when you go to high resolution (even higher
than HD) it may also become a limitation. 
Roni Even
As individual

> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On
> Behalf Of tom harper
> Sent: Tuesday, April 24, 2012 3:19 PM
> To: [e-mail]
> Subject: Re: [payload] VP8 payload, decoder processing capabilities
> (was Re: [rtcweb] Resolution negotiation - a contribution)
> 
> I think the max-fps might be what is needed.
> 
> A general frame rate display limitation with a device ( i.e. it can
> only do 15, 30 fps) such as a phone could just use frame rate the way
> Harald described on the media line- the case of I can do VGA-30 (but
> not 120
> fps) and I can do 720p-30.
> 
> What this would not handle would be if there is some other overall
> processing cap that is codec specific, and your cpu/hardware could do
> 720p 30 and 1080p 15, but 60 is your max due to some programmatic
> limitation or choice.
> 
> You could in theory do it with the max-fps  parameter in the following
> manner- Using maxmbps/maxfs, infer the maximum codec frame rate at the
> maximum resolution to be max-mbps/max-fs.  Then you need a max-fps
> parameter to cap your max framerate at lower resolutions.  You could
> infer the max framerate in between by calculating the max-mbps/max-fs
> ratio, assuming it is linear.
> 
> You could make max-fps optional by having a default max-fps = max-
> mbps/max-fs.
> 
> The above should in theory cover multiple video streams also, where
> each stream's capability could be shaped.
> 
> If there is a more complicated use case you would have to probably
> specify exact resolution/frame rate combos, it would be optional, so
> probably doesn't need to be specified here.
> 
> Tom
> 
> 
> On 4/17/2012 11:22 AM, Charles Eckel (eckelcu) wrote:
> > If I understand correctly, one concern is that implementations of VP8
> will encounter similar problems to those of H.264 in terms of
> negotiation of decoder capabilities. H.264 allows you to negotiate max-
> mbps and max-fs for the maximum macroblocks per second and maximum
> frame size, respectively. In theory, this should be sufficient.
> Unfortunately, many real world implementations have limitations in
> terms of max frame rate as well. While they can receive 1080p at 30
> fps, they cannot receive 360p at 60 fps, or 180p at 172 fps. These
> combinations fall well within the max-mbps and max-fs limits, but they
> still are not supported due to limitations of implementations.
> > The proposal of the max-fps parameter for H.264
> (http://tools.ietf.org/html/draft-kristensen-avt-rtp-h241param-01) is a
> response to this. It is a work in progress.
> > It seems something similar may be necessary in practice for VP8.
> > I agree with Tom that the existing max-framerate at the media line
> level is not sufficient.
> >
> > Cheers,
> > Charles
> >
> >> -----Original Message-----
> >> From: [e-mail] [mailto:[e-mail]] On
> >> Behalf Of tom harper
> >> Sent: Tuesday, April 17, 2012 12:47 AM
> >> To: Roni Even
> >> Cc: [e-mail]
> >> Subject: Re: [payload] VP8 payload, decoder processing capabilities
> >> (was Re: [rtcweb] Resolution negotiation - a contribution)
> >>
> >> I understand where Harald is coming from, VP8 doesn't have the same
> >> computational jump that h264 has between cavlc and cabac at the
> >> decoder.   With h264 this is important, as there are a bunch of
> devices
> >> that only do the former.  I don't think this applies to vp8 that I
> >> know of.
> >>
> >> So you don't need a level equivalent.  However, I don't know that
> the
> >> standards proposed by Harald do everything either.
> >>
> >> Unless I misread, I think rfc 4566 a=framerate is intended to be
> >> defined at the media level for all video media-  For devices with a
> >> general frame rate cap this is fine, but often you instead want a
> >> codec specific max frame rate, due to the varying complexities of
> >> differing codecs.
> >> Also, it isn't clear to me from looking at the imageattr spec that
> >> you can easily specify a range of resolutions other than listing
> them
> >> out in verbose detail.  I don't think that is ideal either.  And I
> >> might ultimately have a frame rate constraint at a higher resolution
> >> that I don't have at a lower one.  So that wouldn't be clear using
> >> imageattr.
> >>
> >> For the cases a=framerate doesn't handle, there still needs to be
> the
> >> ability to negotiate a codec specific preferred max display size, as
> >> well as the codec specific computational limit of decoding,.
> >>
> >> So I would vote for a solution like maxmbps- the max complexity is
> >> then understood.  Then I guess you would need a maxfs equivalent
> >> also.  Then you could just allow the application to let the frame
> >> rate vary as needed.
> >>
> >> as me,
> >>
> >> Tom
> >>
> >> On 4/16/2012 11:48 PM, Roni Even wrote:
> >>> Hi,
> >>> I think that one point that is missing here is if the issue is only
> >> for the
> >>> receiver capability or request or if there is a need to specify
> also
> >> the
> >>> encoder capabilities.
> >>> In H.264 (RFC6184) the level parameter signal the encoding and
> >> decoding
> >>> capability and the answers can signal a lower level (If asymmetry
> is
> >>> supported). The level provides some maximum on what the receiver
> can
> >>> request.
> >>> I think that the image attribute does not provide this
> functionality
> >> and I
> >>> am not sure if this is the meaning of the SamplesPerSecond
> parameter.
> >>>
> >>> I am keeping the discussion in payload WG
> >>>
> >>> Roni Even
> >>> As individual
> >>>
> >>>> -----Original Message-----
> >>>> From: [e-mail] [mailto:[e-mail]]
> On
> >>>> Behalf Of Stephan Wenger
> >>>> Sent: Tuesday, April 17, 2012 12:40 AM
> >>>> To: Harald Alvestrand
> >>>> Cc: [e-mail]; [e-mail]
> >>>> Subject: [payload] VP8 payload, decoder processing capabilities
> >>>> (was
> >>>> Re: [rtcweb] Resolution negotiation - a contribution)
> >>>>
> >>>> Hi all,
> >>>>
> >>>> For context: Harald and myself have been at odds for a while now
> >> about
> >>>> the lack of support for a code point in the VP8 payload that can
> be
> >>>> used to negotiate a maximum decoder/bitstream complexity.
> >>>> Specifically, Harald (and other VP8 payload folks) suggested that
> >>>> generic mechanisms, such as the a=framerate attribute of RFC4566
> in
> >>>> conjunction with the picture size aspect of the imageattr of RFC
> >> 6236
> >>>> can be used, at least in the rtcweb context.  However, as far as I
> >>>> understood our argument, these two mechanisms in combination are
> >>>> not meant as a limit for decoder complexity (in terms of
> >>>> samples/sec processing rate), but rather as an indication, from
> >>>> receiver to
> >> sender,
> >>>> of an upper bound of what is "useful to send".
> >>>> See the email below.  To me, it's quite obvious that an indication
> >> of
> >>>> "useful to send" includes "my decoder can handle this"; however,
> it
> >>>> could be more restrictive in that factors other than decoder
> >> horsepower
> >>>> could also be at play, such as screen size, user interface
> >>>> settings, and so on.
> >>>>
> >>>> I believe that the combination of what can be signaled using the
> >> above
> >>>> mechanisms should be sufficient for rtcweb.  However, I also
> >>>> believe that it is insufficient for general purpose use, mostly
> >>>> because it requires the support of RFC 6236, which is not exactly
> a
> >>>> widely deployed technology.
> >>>> Further, the a=framerate attribute is not a particularly useful
> >>>> attribute these days anymore, because variable frame rates, at
> >>>> least for software encoding/decoding, are the norm.
> >>>>
> >>>> In previous posts on the payload list (in response to the VP8
> >> payload
> >>>> WGLC), I have commented on the practical shortcomings of the (lack
> >> of)
> >>>> complexity negotiation, and suggested that this needs to be fixed.
> >>>>
> >>>> Two options:
> >>>>
> >>>> 1) codify Harald's mechanism (based on a=framerate and imageattr
> in
> >> the
> >>>> VP8 payload draft, at MUST strength.  "In a declarative context, a
> >>>> prospective media sender supporting this (VP8 payload)
> >>>> specification MUST support RFC 4566 a=framerate and RFC6236
> >>>> imageattr, and MUST include code points according to both
> >>>> mechanisms to identify the properties of the media stream.  In an
> >>>> offer-answer context, both offerer and answerer receiver
> supporting
> >>>> this VP8 payload
> >> specification
> >>>> MUST support a=framertate and imageattr, and MUST include both in
> >> their
> >>>> respective offer/answer messages, so to identify an operation
> point
> >>>> that will not overload the media decoder's capabilities.
> >>>>
> >>>> The issue with this approach, IMO, is that we are dealing here
> with
> >>>> three individual code points (framerate, horizontal and vertical
> >>>> picture size), where a single code point ought to be sufficient
> for
> >>>> determining whether a décor is capable of decoding a stream, at
> >> least
> >>>> from a complexity viewpoint).
> >>>>
> >>>> 2) include, in the V8 payload, a negotiable SDP code point
> >> indicating
> >>>> the complexity of a stream, in units of samples per second
> >> processing
> >>>> requirements or a derivative thereof (such as: levels as used in
> >>>> the MPEG world).  For example, the VP8 payload could include a
> >>>> single, optional, negotiable parameter "SamplePerSecond".  If
> >> SamplePerSecond
> >>>> were absent in the SDP, a value of xxxxx must be inferred.  (a
> >> sensible
> >>>> value for xxxxx could be, for example 9216000, which is the number
> >> of
> >>>> samples per second for VGA resolution at 30 Hz).  If
> >>>> SamplePerSecond
> >> is
> >>>> present in a declarative context, it indicates the minimum
> >> processing
> >>>> requirements a decoder must support in order to successfully
> decode
> >> the
> >>>> stream.  In a symmetric offer-answer context, SamplePerSecond can
> >>>> be used to "dial down"
> >>>> the complexity of the stream to a value that both encoder and
> >> decoder
> >>>> can support.
> >>>>
> >>>> My preference is obviously the second proposal, but I'm willing to
> >> help
> >>>> fleshing out either or both of them, just not today :-)
> >>>>
> >>>> Regards,
> >>>> Stephan
> >>>>
> >>>>
> >>>>
> >>>> On 4.13.2012 00:45 , "Harald Alvestrand"<[e-mail]>
> >> wrote:
> >>>>> On 04/12/2012 11:13 PM, Stephan Wenger wrote:
> >>>>>> On 4.12.2012 12:08 , "Harald Alvestrand"<[e-mail]>
> >>>> wrote:
> >>>>>>> On 04/12/2012 08:19 PM, Stephan Wenger wrote:
> >>>>>>>> Hi Harald,
> >>>>>>>> Thanks for this strawman.  I believe it should work, but I
> fail
> >> to
> >>>>>>>> see how a two dimensional negotiation requirement (negotiating
> >> max
> >>>>>>>> values for framerate and image size--which, in turn, also has
> >>>>>>>> two-dimensional
> >>>>>>>> properties) leads to better interop than a one dimensional
> >>>>>>>> negotiation (pixels per second processing requirement).
> >>>>>>> Stephan,
> >>>>>>>
> >>>>>>> I do not see this (or the requirement from the use-cases
> >> document)
> >>>>>>> first  and foremost a decoder complexity negotiation; it is a
> >>>>>>> negotiation of  how much data it is useful to send, given the
> >>>>>>> recipient's intended use  of that data.
> >>>>>> Then such a negotiation should be executed in addition.  Decoder
> >>>>>> cycle requirement do matter in practical implementations.
> >>>>> Feel free to propose language that captures this requirement. As
> >>>> noted,
> >>>>> my I-D fragment doesn't.
> >>>>>
> >>>>>
> >>>> _______________________________________________
> >>>> payload mailing list
> >>>> [e-mail]
> >>>> https://www.ietf.org/mailman/listinfo/payload
> >>> _______________________________________________
> >>> payload mailing list
> >>> [e-mail]
> >>> https://www.ietf.org/mailman/listinfo/payload
> >>>
> >> _______________________________________________
> >> payload mailing list
> >> [e-mail]
> >> https://www.ietf.org/mailman/listinfo/payload
> _______________________________________________
> payload mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/payload