Re: [codec] Summary of test results

Koen Vos <koen.vos@skype.net> Wed, 22 June 2011 13:31 UTC

Return-Path: <koen.vos@skype.net>
X-Original-To: codec@ietfa.amsl.com
Delivered-To: codec@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id F2D1711E813E for <codec@ietfa.amsl.com>; Wed, 22 Jun 2011 06:31:40 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id phJ3CnNjrTMp for <codec@ietfa.amsl.com>; Wed, 22 Jun 2011 06:31:40 -0700 (PDT)
Received: from mx.skype.net (mx.skype.net [78.141.177.88]) by ietfa.amsl.com (Postfix) with ESMTP id 64EAB11E812B for <codec@ietf.org>; Wed, 22 Jun 2011 06:31:38 -0700 (PDT)
Received: from mx.skype.net (localhost [127.0.0.1]) by mx.skype.net (Postfix) with ESMTP id 06FD91712; Wed, 22 Jun 2011 15:31:37 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=skype.net; h=date:from:to :cc:message-id:in-reply-to:subject:mime-version:content-type: content-transfer-encoding; s=mx; bh=AxWe/Kw45INVDovIA59nQoj3ncU= ; b=rmAuLfHkv1vsPtj01HY9hrStj9DF5XWGJFyaY0nF7N51/uRsKERs6HTzuVAd NVPVNC5nic8ziyHoQ5TFPcRbbSU6oE4m5oYzwENewliQWspK0OXyk/Vrq5KHG30U 3TOpXB0A4STK9sf0drKwum303hJz0+Wpn/C+W8sFs1XZJUU=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=skype.net; h=date:from:to:cc :message-id:in-reply-to:subject:mime-version:content-type: content-transfer-encoding; q=dns; s=mx; b=P5ucvTzs26GV2OOIte+Kwj i5VN2jrO1SowSEmVF9jTGkZ9Wwboss9JA7dWoJAfQ2+vQG6j58QANhOwtp9+2Ljy UtoHRY59tLtvZ7b3GuklcFqEpUSNQEyduIZNAQUZNDgMOUzkeDDWxVcjgowEZ60v Un06bYw7uHImAWOE1EEbU=
Received: from zimbra.skype.net (zimbra.skype.net [78.141.177.82]) by mx.skype.net (Postfix) with ESMTP id 056B31700; Wed, 22 Jun 2011 15:31:37 +0200 (CEST)
Received: from localhost (localhost [127.0.0.1]) by zimbra.skype.net (Postfix) with ESMTP id D14533507EF1; Wed, 22 Jun 2011 15:31:36 +0200 (CEST)
X-Virus-Scanned: amavisd-new at lu2-zimbra.skype.net
Received: from zimbra.skype.net ([127.0.0.1]) by localhost (zimbra.skype.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CcGlcXq1X6mH; Wed, 22 Jun 2011 15:31:35 +0200 (CEST)
Received: from zimbra.skype.net (lu2-zimbra.skype.net [78.141.177.82]) by zimbra.skype.net (Postfix) with ESMTP id 06ED83507F39; Wed, 22 Jun 2011 15:31:34 +0200 (CEST)
Date: Wed, 22 Jun 2011 15:31:33 +0200 (CEST)
From: Koen Vos <koen.vos@skype.net>
To: Erik Norvell <erik.norvell@ericsson.com>
Message-ID: <1826386229.2244245.1308749493214.JavaMail.root@lu2-zimbra>
In-Reply-To: <027A93CE4A670242BD91A44E37105AEF18634B6957@ESESSCMS0351.eemea.ericsson.se>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Originating-IP: [69.181.192.115]
X-Mailer: Zimbra 6.0.9_GA_2686 (ZimbraWebClient - FF3.0 (Win)/6.0.9_GA_2686)
Cc: codec@ietf.org
Subject: Re: [codec] Summary of test results
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 22 Jun 2011 13:31:41 -0000

Hi Erik,

> However, the only way to make correct statements about the performance 
> of the final Opus codec is to test this final codec.

We took the MPEG approach to standardization (like with MP3/AAC), which means that the codec is defined by its bit-stream rather than its bit-exact behavior.  For this reason, the version tested from February is the final Opus.  Besides, the encoder hasn't actually changed in any meaningful way since February.

best,
koen.


----- Original Message -----
From: "Erik Norvell" <erik.norvell@ericsson.com>
To: "Jean-Marc Valin" <jean-marc.valin@octasic.com>
Cc: codec@ietf.org
Sent: Wednesday, June 22, 2011 2:00:10 AM
Subject: Re: [codec] Summary of test results

> -----Original Message-----
> From: Jean-Marc Valin [mailto:jean-marc.valin@octasic.com]
> Sent: den 21 juni 2011 21:40
> To: Erik Norvell
> Cc: codec@ietf.org
> Subject: Re: [codec] Summary of test results
> 
> On 11-06-21 09:04 AM, Erik Norvell wrote:
> > Thank you for compiling this summary of pre-Opus tests. It should 
> > definitely help in designing the listening test on the final Opus.
> 
> Just to clarify, the Opus bit-stream *is* final and, as far as these 
> tests (for both speech and music) are concerned, has been since 
> February.
> The latest draft also has the final stereo bit-stream for voice, but 
> all the rest is long frozen.
> 

There are a number of tests which are older than that which are still referenced when making statements about Opus performance.
In addition, a frozen bit-stream is not equal to frozen quality. If the codec itself is still permitted to change its quality may be affected.


> > One comment to section 3: "While Opus has evolved since these tests 
> > were conducted, the results should be considered as a
> _lower bound_ on
> > the quality of the final codec."
> >
> > I would like to think that the sum is always greater than
> it's parts,
> > but it is definitely possible to make something worse by
> working on it.
> > Hence, statements about Opus performance must be based on
> tests made
> > on the final codec.
> 
> Of course it's not a guarantee, but there's definitely value in those 
> tests results in that it's unlikely that everything always worked fine 
> and then we just screwed everything up at the end (if that was the 
> case we would have realised it in the other tests).
> 

I agree the tests are valuable as quality indicators for the codec by the time they were conducted. However, the only way to make correct statements about the performance of the final Opus codec is to test this final codec. To deduce that performance from tests of previous versions is bound to include some amount of speculation.

Best,
Erik
_______________________________________________
codec mailing list
codec@ietf.org
https://www.ietf.org/mailman/listinfo/codec