Re: [codec] Summary of test results

"Christian Hoene" <hoene@uni-tuebingen.de> Sun, 26 June 2011 11:01 UTC

Return-Path: <hoene@uni-tuebingen.de>
X-Original-To: codec@ietfa.amsl.com
Delivered-To: codec@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5EFDB1F0C3D for <codec@ietfa.amsl.com>; Sun, 26 Jun 2011 04:01:19 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.795
X-Spam-Level:
X-Spam-Status: No, score=-4.795 tagged_above=-999 required=5 tests=[AWL=1.454, BAYES_00=-2.599, HELO_EQ_DE=0.35, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id K-DWcJnuvbHV for <codec@ietfa.amsl.com>; Sun, 26 Jun 2011 04:01:18 -0700 (PDT)
Received: from mx06.uni-tuebingen.de (mx06.uni-tuebingen.de [134.2.3.3]) by ietfa.amsl.com (Postfix) with ESMTP id 690D81F0C37 for <codec@ietf.org>; Sun, 26 Jun 2011 04:01:18 -0700 (PDT)
Received: from hoeneT60 (u-173-c044.cs.uni-tuebingen.de [134.2.173.44]) (authenticated bits=0) by mx06.uni-tuebingen.de (8.13.6/8.13.6) with ESMTP id p5QB1FAD019812 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Sun, 26 Jun 2011 13:01:16 +0200
From: "Christian Hoene" <hoene@uni-tuebingen.de>
To: "'Koen Vos'" <koen.vos@skype.net>, "'Erik Norvell'" <erik.norvell@ericsson.com>
References: <027A93CE4A670242BD91A44E37105AEF18634B6957@ESESSCMS0351.eemea.ericsson.se> <1826386229.2244245.1308749493214.JavaMail.root@lu2-zimbra>
In-Reply-To: <1826386229.2244245.1308749493214.JavaMail.root@lu2-zimbra>
Date: Sun, 26 Jun 2011 13:01:16 +0200
Organization: =?us-ascii?Q?Universitat_Tubingen?=
Message-ID: <002501cc33f0$5f1c2770$1d547650$@uni-tuebingen.de>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Outlook 14.0
Thread-Index: AQIdJU09Y8GgTD3xSsk7FTHsxVlpgpQtlNRA
Content-Language: de
X-AntiVirus: NOT checked by Avira MailGate (version: 3.1.2; host: mx06)
Cc: codec@ietf.org
Subject: Re: [codec] Summary of test results
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 26 Jun 2011 11:01:19 -0000

Hi,

Erik comments are valid. We should make some tests on the final codec
specification before WGLC. I mean, in the latest listening tests which used
Opus taken from GIT February 16th, 2011, I found the statement that "with
lower bitrates the codec crashes".  

I mean, just checking the implementation against major programming mistakes
is not a bad think. And this should not be made by the codec designers
themselves.

With best regards,

 Christian

> -----Original Message-----
> From: codec-bounces@ietf.org [mailto:codec-bounces@ietf.org] On Behalf
> Of Koen Vos
> Sent: Wednesday, June 22, 2011 3:32 PM
> To: Erik Norvell
> Cc: codec@ietf.org
> Subject: Re: [codec] Summary of test results
> 
> Hi Erik,
> 
> > However, the only way to make correct statements about the performance
> > of the final Opus codec is to test this final codec.
> 
> We took the MPEG approach to standardization (like with MP3/AAC), which
> means that the codec is defined by its bit-stream rather than its
bit-exact
> behavior.  For this reason, the version tested from February is the final
Opus.
> Besides, the encoder hasn't actually changed in any meaningful way since
> February.
> 
> best,
> koen.
> 
> 
> ----- Original Message -----
> From: "Erik Norvell" <erik.norvell@ericsson.com>
> To: "Jean-Marc Valin" <jean-marc.valin@octasic.com>
> Cc: codec@ietf.org
> Sent: Wednesday, June 22, 2011 2:00:10 AM
> Subject: Re: [codec] Summary of test results
> 
> > -----Original Message-----
> > From: Jean-Marc Valin [mailto:jean-marc.valin@octasic.com]
> > Sent: den 21 juni 2011 21:40
> > To: Erik Norvell
> > Cc: codec@ietf.org
> > Subject: Re: [codec] Summary of test results
> >
> > On 11-06-21 09:04 AM, Erik Norvell wrote:
> > > Thank you for compiling this summary of pre-Opus tests. It should
> > > definitely help in designing the listening test on the final Opus.
> >
> > Just to clarify, the Opus bit-stream *is* final and, as far as these
> > tests (for both speech and music) are concerned, has been since
> > February.
> > The latest draft also has the final stereo bit-stream for voice, but
> > all the rest is long frozen.
> >
> 
> There are a number of tests which are older than that which are still
> referenced when making statements about Opus performance.
> In addition, a frozen bit-stream is not equal to frozen quality. If the
codec
> itself is still permitted to change its quality may be affected.
> 
> 
> > > One comment to section 3: "While Opus has evolved since these tests
> > > were conducted, the results should be considered as a
> > _lower bound_ on
> > > the quality of the final codec."
> > >
> > > I would like to think that the sum is always greater than
> > it's parts,
> > > but it is definitely possible to make something worse by
> > working on it.
> > > Hence, statements about Opus performance must be based on
> > tests made
> > > on the final codec.
> >
> > Of course it's not a guarantee, but there's definitely value in those
> > tests results in that it's unlikely that everything always worked fine
> > and then we just screwed everything up at the end (if that was the
> > case we would have realised it in the other tests).
> >
> 
> I agree the tests are valuable as quality indicators for the codec by the
time
> they were conducted. However, the only way to make correct statements
> about the performance of the final Opus codec is to test this final codec.
To
> deduce that performance from tests of previous versions is bound to
include
> some amount of speculation.
> 
> Best,
> Erik
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec