Re: [rtcweb] More H.264 vs VP8 tests

Stefan Håkansson LK <> Tue, 25 June 2013 14:56 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 0ECBD21F9B06 for <>; Tue, 25 Jun 2013 07:56:52 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -3.82
X-Spam-Status: No, score=-3.82 tagged_above=-999 required=5 tests=[AWL=-1.521, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id n8unK7xP12yN for <>; Tue, 25 Jun 2013 07:56:47 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id 1466211E8132 for <>; Tue, 25 Jun 2013 07:56:42 -0700 (PDT)
X-AuditID: c1b4fb38-b7f476d0000049c9-bf-51c9af9fe7ba
Received: from (Unknown_Domain []) by (Symantec Mail Security) with SMTP id 8A.4F.18889.F9FA9C15; Tue, 25 Jun 2013 16:56:32 +0200 (CEST)
Received: from ([]) by ([]) with mapi id 14.02.0328.009; Tue, 25 Jun 2013 16:56:31 +0200
From: =?iso-8859-1?Q?Stefan_H=E5kansson_LK?= <>
To: Harald Alvestrand <>
Thread-Topic: [rtcweb] More H.264 vs VP8 tests
Thread-Index: Ac5vQ4bE4hnmu5ERSv6YOg93i7vBVw==
Date: Tue, 25 Jun 2013 14:56:30 +0000
Message-ID: <>
References: <> <> <> <>
Accept-Language: en-US
Content-Language: en-US
x-originating-ip: []
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrELMWRmVeSWpSXmKPExsUyM+Jvre6C9ScDDZasZLI41tfFZrH2Xzu7 A5PHlQlXWD2WLPnJFMAUxWWTkpqTWZZapG+XwJVx4uxWxoKVWhVTOv6zNDB+Vexi5OSQEDCR WNv8gBnCFpO4cG89WxcjF4eQwFFGibfLVzCCJIQEFjFKnGpQBbHZBAIltu5bwAZiiwjoSDzc 38AEYjMLqEvcWXyOHcQWFtCVaDi+BqpGT2Lh8xcsMPbu/R/BlrEIqEq0PbwOFucV8JX4vbGT HWLxb0aJ3rk3wAYxAl30/dQaqAXiEreezGeCuFRAYsme81BXi0q8fPyPFcJWlNh5tp0Zol5P 4sbUKWwQtrbEsoWvmSGWCUqcnPmEZQKj6CwkY2chaZmFpGUWkpYFjCyrGDmKU4uTctONDDYx AuPh4JbfFjsYL/+1OcQozcGiJM776dSuQCGB9MSS1OzU1ILUovii0pzU4kOMTBycUg2MzLbP Vs/M+L7A9r+BTr+mx5bil9M3rHPO/rZP5Lwdx8+8XYpsPh+bWVosVp1tXK3EvXPrw/6G/Jx9 6rudfkRtqFwQIrl/wdP4A9N0zhqmv3FMsF7nvvVmYcbSh9W1To0RXkK8mpsD2I/Yli7bOJc9 Xq2oRm6Wnnska2v5DdkNgn4ijm9mOPxTYinOSDTUYi4qTgQAkcNmrVUCAAA=
Cc: "" <>
Subject: Re: [rtcweb] More H.264 vs VP8 tests
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Tue, 25 Jun 2013 14:56:52 -0000

On 2013-06-25 00:00, Harald Alvestrand wrote:
> First of all, Bo, thanks for running the tests, and for making the
> results fully open!
> On 06/24/2013 01:33 PM, Stefan Håkansson LK wrote:
>> On 2013-06-22 18:54, Leon Geyser wrote:
>>> Hi Bo,
>>> Thanks for the additional testing done.
>>> I am not 100% why we have to test with fixed qp-levels when WebRTC is
>>> going to be used over the internet. I would assume that WebRTC would be
>>> used over the internet...
>>> My understanding of rate-control might be completely wrong. Why test
>>> modes that might not even be used in real-world scenarios?
>> Hi Leon (responding for Bo who is enjoying his vacation),
>> you are absolutely right, these codecs are going to be used with
>> rate-controllers when deployed in the real world. However, we still do
>> not think it is a good idea having rate control turned on when comparing
>> them. The reason is that the rate controller influences the quality so
>> much that it is very hard to get any comparable signal if you include
>> them. Let me give you one example; let's say that you use the same
>> codec, but two different rate controllers. Rate control A is very
>> liberal with its bits in the beginning of the sequence, whereas rate
>> control B is very frugal. On a video conferencing type sequence, where
>> the camera is static and not much is happening, rate control A will win
>> by a huge margin over rate control B. The reason is that the first frame
>> will be encoded very well and this data can be used during the remainder
>> of the sequence. Rate control B will not be able to stand a chance, even
>> though it has more bits for the remaining images.  And this is with the
>> same video codec!
> This would seem to me to be an argument for including the rate
> control.... what we are looking for is the codec where people can
> demonstrate the best performance for our specific situations, not the
> codec that is best able to fulfil a set of conditions that are
> inappropriate for Real Life.
> For the example you give, the bit division between the I-frame/key-frame
> and the subsequent frames is not just influenced by the chosen Q value -
> it's also influenced by codec design.
> What will happen in a video conference scenario if someone uses this
> rate packing too much is that the I-frame takes a long time to transmit,
> because the available bit rate is in the ramp-up phase; if we wanted to
> make realistic scenario settings, we should probably draw constraints
> around the delay in delivering the second frame under specified
> bandwidth conditions.
> But on the other hand, people may find a delay to the second frame quite
> acceptable, and your "rate control A" is indeed the one that is most
> reasonable to use.

I think that the rate control should be omitted from this kind of 
comparisons since it adds a lot of noise to the results and since - as 
far as I have understood - the rate controller is rather independent of 
the codec (you could use very similar rate controllers for VP8 and h.264).

But I agree to the main point you make further below, so perhaps this is 
a debate that can be saved for another discussion.

>> We could of course try to make sure that the two rate controllers have
>> similar settings; however, since they are not identical they cannot be
>> made to work exactly the same. Even small changes in rate control
>> settings could have a huge effect on the end result.
>> Rather than trying to make two pieces of software work the same, it is
>> much easier to just shut off the rate controllers. This gives a direct
>> measurement of the relative strengths of the codecs. This is also the
>> reason why organizations such as JC-TVC have used this method to compare
>> their codecs.
> And it's going to be discussed there.... I'll be late for the Berlin
> IETF meeting because I have to be at the MPEG meeting in Vienna to
> present Google's formal submission of VP8 to MPEG. The use or non-use of
> variable QP will probably be an item of debate .... again.
> The IETF has not started down the path of defining test conditions under
> which it wishes to measure codec performance; I think we do have rough
> consensus that the technical differences are not so great as to make a
> decisive difference between the proposed codecs - while the IPR
> differences are all too well known to us, and do matter enough to make a
> decisive difference to a lot of us.

I agree, and our tests underline, that the performance differences are 
not so great as to make any decisive difference between the VP8 and 
h.264 baseline - their performance is very similar.

>                  Harald
> _______________________________________________
> rtcweb mailing list