Re: [codec] Summary of test results

Jean-Marc Valin <jean-marc.valin@octasic.com> Tue, 21 June 2011 19:40 UTC

Return-Path: <jean-marc.valin@octasic.com>
X-Original-To: codec@ietfa.amsl.com
Delivered-To: codec@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2ABCE1F0C38 for <codec@ietfa.amsl.com>; Tue, 21 Jun 2011 12:40:49 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5XQd8xubF6Km for <codec@ietfa.amsl.com>; Tue, 21 Jun 2011 12:40:46 -0700 (PDT)
Received: from toroondcbmts04-srv.bellnexxia.net (toroondcbmts04.bellnexxia.net [207.236.237.38]) by ietfa.amsl.com (Postfix) with ESMTP id DAD041F0C71 for <codec@ietf.org>; Tue, 21 Jun 2011 12:40:22 -0700 (PDT)
Received: from toip57-bus.srvr.bell.ca ([67.69.240.184]) by toroondcbmts04-srv.bellnexxia.net (InterMail vM.8.00.01.00 201-2244-105-20090324) with ESMTP id <20110621194021.RWEN11416.toroondcbmts04-srv.bellnexxia.net@toip57-bus.srvr.bell.ca>; Tue, 21 Jun 2011 15:40:21 -0400
Received: from toip36-bus.srvr.bell.ca ([67.69.240.37]) by toip57-bus.srvr.bell.ca with ESMTP; 21 Jun 2011 15:40:17 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: Av0EANbtAE7PPaAN/2dsb2JhbABUpnd3iHO/e4YqBJZICYs3
Received: from mail.octasic.com ([207.61.160.13]) by toip36-bus.srvr.bell.ca with ESMTP; 21 Jun 2011 15:40:17 -0400
Received: from [10.100.50.90] (10.100.50.90) by MAIL2.octasic.com (10.100.10.44) with Microsoft SMTP Server (TLS) id 14.1.270.1; Tue, 21 Jun 2011 15:40:16 -0400
Message-ID: <4E00F3A0.5030800@octasic.com>
Date: Tue, 21 Jun 2011 15:40:16 -0400
From: Jean-Marc Valin <jean-marc.valin@octasic.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: Erik Norvell <erik.norvell@ericsson.com>
References: <4DFA5106.1040903@octasic.com> <027A93CE4A670242BD91A44E37105AEF18634B66E9@ESESSCMS0351.eemea.ericsson.se>
In-Reply-To: <027A93CE4A670242BD91A44E37105AEF18634B66E9@ESESSCMS0351.eemea.ericsson.se>
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit
X-Originating-IP: [10.100.50.90]
Cc: "codec@ietf.org" <codec@ietf.org>
Subject: Re: [codec] Summary of test results
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 21 Jun 2011 19:40:49 -0000

On 11-06-21 09:04 AM, Erik Norvell wrote:
> Thank you for compiling this summary of pre-Opus tests. It should
> definitely help in designing the listening test on the final Opus.

Just to clarify, the Opus bit-stream *is* final and, as far as these
tests (for both speech and music) are concerned, has been since February.
The latest draft also has the final stereo bit-stream for voice, but all
the rest is long frozen.

> One comment to section 3: "While Opus has evolved since these tests were
> conducted, the results should be considered as a _lower bound_ on the
> quality of the final codec."
>
> I would like to think that the sum is always greater than it's parts,
> but it is definitely possible to make something worse by working on it.
> Hence, statements about Opus performance must be based on tests made on
> the final codec.

Of course it's not a guarantee, but there's definitely value in those tests
results in that it's unlikely that everything always worked fine and then 
we just screwed everything up at the end (if that was the case we would 
have realised it in the other tests).

Cheers,

	Jean-Marc


> Best regards, Erik
>
>> -----Original Message----- From: codec-bounces@ietf.org
>> [mailto:codec-bounces@ietf.org] On Behalf Of Jean-Marc Valin Sent: den
>> 16 juni 2011 20:53 To: codec@ietf.org Subject: [codec] Summary of test
>> results
>>
>> Hi,
>>
>> We've collected all the test results we could find for Opus (and
>> pre-Opus) and how they relate to the requirements. We summarized it
>> all in the following draft:
>> http://tools.ietf.org/html/draft-valin-codec-results-00
>>
>> Cheers,
>>
>> Jean-Marc _______________________________________________ codec
>> mailing list codec@ietf.org
>> https://www.ietf.org/mailman/listinfo/codec