Re: [rtcweb] More H.264 vs VP8 tests

Stefan Håkansson LK <stefan.lk.hakansson@ericsson.com> Tue, 25 June 2013 14:56 UTC

Return-Path: <stefan.lk.hakansson@ericsson.com>
X-Original-To: rtcweb@ietfa.amsl.com
Delivered-To: rtcweb@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0ECBD21F9B06 for <rtcweb@ietfa.amsl.com>; Tue, 25 Jun 2013 07:56:52 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.82
X-Spam-Level:
X-Spam-Status: No, score=-3.82 tagged_above=-999 required=5 tests=[AWL=-1.521, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id n8unK7xP12yN for <rtcweb@ietfa.amsl.com>; Tue, 25 Jun 2013 07:56:47 -0700 (PDT)
Received: from sesbmg20.ericsson.net (sesbmg20.ericsson.net [193.180.251.56]) by ietfa.amsl.com (Postfix) with ESMTP id 1466211E8132 for <rtcweb@ietf.org>; Tue, 25 Jun 2013 07:56:42 -0700 (PDT)
X-AuditID: c1b4fb38-b7f476d0000049c9-bf-51c9af9fe7ba
Received: from ESESSHC020.ericsson.se (Unknown_Domain [153.88.253.125]) by sesbmg20.ericsson.net (Symantec Mail Security) with SMTP id 8A.4F.18889.F9FA9C15; Tue, 25 Jun 2013 16:56:32 +0200 (CEST)
Received: from ESESSMB209.ericsson.se ([169.254.9.6]) by ESESSHC020.ericsson.se ([153.88.183.78]) with mapi id 14.02.0328.009; Tue, 25 Jun 2013 16:56:31 +0200
From: Stefan Håkansson LK <stefan.lk.hakansson@ericsson.com>
To: Harald Alvestrand <harald@alvestrand.no>
Thread-Topic: [rtcweb] More H.264 vs VP8 tests
Thread-Index: Ac5vQ4bE4hnmu5ERSv6YOg93i7vBVw==
Date: Tue, 25 Jun 2013 14:56:30 +0000
Message-ID: <1447FA0C20ED5147A1AA0EF02890A64B1C302C38@ESESSMB209.ericsson.se>
References: <BBE9739C2C302046BD34B42713A1E2A22DECC12F@ESESSMB105.ericsson.se> <CAGgHUiRmAu_8R53qZnmTFHE3016sFSD+FtZ=zTG1wjBjRxL3MA@mail.gmail.com> <1447FA0C20ED5147A1AA0EF02890A64B1C3020B6@ESESSMB209.ericsson.se> <51C8C16B.4070405@alvestrand.no>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [153.88.183.16]
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrELMWRmVeSWpSXmKPExsUyM+Jvre6C9ScDDZasZLI41tfFZrH2Xzu7 A5PHlQlXWD2WLPnJFMAUxWWTkpqTWZZapG+XwJVx4uxWxoKVWhVTOv6zNDB+Vexi5OSQEDCR WNv8gBnCFpO4cG89WxcjF4eQwFFGibfLVzCCJIQEFjFKnGpQBbHZBAIltu5bwAZiiwjoSDzc 38AEYjMLqEvcWXyOHcQWFtCVaDi+BqpGT2Lh8xcsMPbu/R/BlrEIqEq0PbwOFucV8JX4vbGT HWLxb0aJ3rk3wAYxAl30/dQaqAXiEreezGeCuFRAYsme81BXi0q8fPyPFcJWlNh5tp0Zol5P 4sbUKWwQtrbEsoWvmSGWCUqcnPmEZQKj6CwkY2chaZmFpGUWkpYFjCyrGDmKU4uTctONDDYx AuPh4JbfFjsYL/+1OcQozcGiJM776dSuQCGB9MSS1OzU1ILUovii0pzU4kOMTBycUg2MzLbP Vs/M+L7A9r+BTr+mx5bil9M3rHPO/rZP5Lwdx8+8XYpsPh+bWVosVp1tXK3EvXPrw/6G/Jx9 6rudfkRtqFwQIrl/wdP4A9N0zhqmv3FMsF7nvvVmYcbSh9W1To0RXkK8mpsD2I/Yli7bOJc9 Xq2oRm6Wnnska2v5DdkNgn4ijm9mOPxTYinOSDTUYi4qTgQAkcNmrVUCAAA=
Cc: "rtcweb@ietf.org" <rtcweb@ietf.org>
Subject: Re: [rtcweb] More H.264 vs VP8 tests
X-BeenThere: rtcweb@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <rtcweb.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/rtcweb>
List-Post: <mailto:rtcweb@ietf.org>
List-Help: <mailto:rtcweb-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/rtcweb>, <mailto:rtcweb-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 25 Jun 2013 14:56:52 -0000

On 2013-06-25 00:00, Harald Alvestrand wrote:
> First of all, Bo, thanks for running the tests, and for making the
> results fully open!
>
> On 06/24/2013 01:33 PM, Stefan Håkansson LK wrote:
>> On 2013-06-22 18:54, Leon Geyser wrote:
>>> Hi Bo,
>>>
>>> Thanks for the additional testing done.
>>>
>>> I am not 100% why we have to test with fixed qp-levels when WebRTC is
>>> going to be used over the internet. I would assume that WebRTC would be
>>> used over the internet...
>>> My understanding of rate-control might be completely wrong. Why test
>>> modes that might not even be used in real-world scenarios?
>> Hi Leon (responding for Bo who is enjoying his vacation),
>>
>> you are absolutely right, these codecs are going to be used with
>> rate-controllers when deployed in the real world. However, we still do
>> not think it is a good idea having rate control turned on when comparing
>> them. The reason is that the rate controller influences the quality so
>> much that it is very hard to get any comparable signal if you include
>> them. Let me give you one example; let's say that you use the same
>> codec, but two different rate controllers. Rate control A is very
>> liberal with its bits in the beginning of the sequence, whereas rate
>> control B is very frugal. On a video conferencing type sequence, where
>> the camera is static and not much is happening, rate control A will win
>> by a huge margin over rate control B. The reason is that the first frame
>> will be encoded very well and this data can be used during the remainder
>> of the sequence. Rate control B will not be able to stand a chance, even
>> though it has more bits for the remaining images.  And this is with the
>> same video codec!
>
> This would seem to me to be an argument for including the rate
> control.... what we are looking for is the codec where people can
> demonstrate the best performance for our specific situations, not the
> codec that is best able to fulfil a set of conditions that are
> inappropriate for Real Life.
>
> For the example you give, the bit division between the I-frame/key-frame
> and the subsequent frames is not just influenced by the chosen Q value -
> it's also influenced by codec design.
>
> What will happen in a video conference scenario if someone uses this
> rate packing too much is that the I-frame takes a long time to transmit,
> because the available bit rate is in the ramp-up phase; if we wanted to
> make realistic scenario settings, we should probably draw constraints
> around the delay in delivering the second frame under specified
> bandwidth conditions.
>
> But on the other hand, people may find a delay to the second frame quite
> acceptable, and your "rate control A" is indeed the one that is most
> reasonable to use.

I think that the rate control should be omitted from this kind of 
comparisons since it adds a lot of noise to the results and since - as 
far as I have understood - the rate controller is rather independent of 
the codec (you could use very similar rate controllers for VP8 and h.264).

But I agree to the main point you make further below, so perhaps this is 
a debate that can be saved for another discussion.

>
>>
>> We could of course try to make sure that the two rate controllers have
>> similar settings; however, since they are not identical they cannot be
>> made to work exactly the same. Even small changes in rate control
>> settings could have a huge effect on the end result.
>>
>> Rather than trying to make two pieces of software work the same, it is
>> much easier to just shut off the rate controllers. This gives a direct
>> measurement of the relative strengths of the codecs. This is also the
>> reason why organizations such as JC-TVC have used this method to compare
>> their codecs.
>
> And it's going to be discussed there.... I'll be late for the Berlin
> IETF meeting because I have to be at the MPEG meeting in Vienna to
> present Google's formal submission of VP8 to MPEG. The use or non-use of
> variable QP will probably be an item of debate .... again.
>
> The IETF has not started down the path of defining test conditions under
> which it wishes to measure codec performance; I think we do have rough
> consensus that the technical differences are not so great as to make a
> decisive difference between the proposed codecs - while the IPR
> differences are all too well known to us, and do matter enough to make a
> decisive difference to a lot of us.

I agree, and our tests underline, that the performance differences are 
not so great as to make any decisive difference between the VP8 and 
h.264 baseline - their performance is very similar.

>
>                  Harald
>
>
> _______________________________________________
> rtcweb mailing list
> rtcweb@ietf.org
> https://www.ietf.org/mailman/listinfo/rtcweb
>