Re: [codec] draft test and processing plan for the IETF Codec

Jean-Marc Valin <jmvalin@jmvalin.ca> Fri, 15 April 2011 10:41 UTC

Return-Path: <jmvalin@jmvalin.ca>
X-Original-To: codec@ietfc.amsl.com
Delivered-To: codec@ietfc.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfc.amsl.com (Postfix) with ESMTP id 0E55DE0778 for <codec@ietfc.amsl.com>; Fri, 15 Apr 2011 03:41:56 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level:
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599]
Received: from mail.ietf.org ([208.66.40.236]) by localhost (ietfc.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UTKzfNDUgVQz for <codec@ietfc.amsl.com>; Fri, 15 Apr 2011 03:41:55 -0700 (PDT)
Received: from relais.videotron.ca (relais.videotron.ca [24.201.245.36]) by ietfc.amsl.com (Postfix) with ESMTP id 1DF01E0689 for <codec@ietf.org>; Fri, 15 Apr 2011 03:41:54 -0700 (PDT)
MIME-version: 1.0
Content-transfer-encoding: 7bit
Content-type: text/plain; charset="ISO-8859-1"; format="flowed"
Received: from [192.168.1.14] ([184.160.206.46]) by VL-MR-MRZ22.ip.videotron.ca (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec 16 2008; 32bit)) with ESMTP id <0LJO00KI8WDTZY40@VL-MR-MRZ22.ip.videotron.ca> for codec@ietf.org; Fri, 15 Apr 2011 06:41:54 -0400 (EDT)
Message-id: <4DA820E3.9000207@jmvalin.ca>
Date: Fri, 15 Apr 2011 06:41:39 -0400
From: Jean-Marc Valin <jmvalin@jmvalin.ca>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
To: Koen Vos <koen.vos@skype.net>
References: <1902603544.172985.1302857051465.JavaMail.root@lu2-zimbra>
In-reply-to: <1902603544.172985.1302857051465.JavaMail.root@lu2-zimbra>
Cc: codec@ietf.org
Subject: Re: [codec] draft test and processing plan for the IETF Codec
X-BeenThere: codec@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Codec WG <codec.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/codec>, <mailto:codec-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/codec>
List-Post: <mailto:codec@ietf.org>
List-Help: <mailto:codec-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/codec>, <mailto:codec-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Apr 2011 10:41:56 -0000

Koen,

The point I was making in the earlier email and that Cullen has stated 
earlier is that we don't even need *any* BT or NWT requirements in this 
document. It will be up to the WG individual participants to decide for 
themselves whether they think it's good to publish Opus, based on the 
information available at the time they make the decision. So Anisse's 
test plan proposal is meant to "gather useful data", not decide on 
whether to publish.

	Jean-Marc

On 11-04-15 04:44 AM, Koen Vos wrote:
> I would also suggest replacing all BT (better than) requirements by NWT (no worse than).
>
> My reasoning is that:
> - The WG never had the goal to be better than other codecs (see charter).
> - Proving to be better can be very hard, especially when several codecs are close to transparent.  To show significance in that case you'd need a vast number of listeners, which makes a test more cumbersome to perform.
>
> best,
> koen.
>
>
> ----- Original Message -----
> From: "Jean-Marc Valin"<jmvalin@jmvalin.ca>
> To: "Jean-Marc Valin"<jmvalin@jmvalin.ca>
> Cc: codec@ietf.org
> Sent: Thursday, April 14, 2011 9:49:58 PM
> Subject: Re: [codec] draft test and processing plan for the IETF Codec
>
> So here's some more specific comments on actual bitrates:
>
> 1) For narrowband Speex, the rates currently listed are 8, 12, 16 kb/s.
> Those should be changed to 8, 11, 15 kb/s to match the actual Speex
> bitrates.
>
> 2) For iLBC, the rates currently listed are 8, 12, 16 kb/s. I think we
> should only use 15.2 kb/s for iLBC. There's another rate, which is 13.33
> kb/s but that's for 30 ms frames so it's not very interesting.
>
> 3) For Speex wideband, the rates currently listed are 12, 24, 32 kb/s. I
> think Speex wideband around 12 kb/s is just crap. Worth testing would be
> 20.6 and 27.8 kb/s.
>
> 4) For super-wideband Speex, I recommend just dumping that. This Speex
> mode was a mistake right from the start and usually has worse quality
> than wideband Speex.
>
> Regarding super-wideband, one thing to keep in mind is that Opus defines
> super-wideband as having a 12 kHz audio bandwidth (24 kHz sampling
> rate). This makes comparisons with other codecs more difficult. The
> rates currently listed for super-wideband are 24, 32, 64 kb/s. I
> recommend running 24 kb/s in super-wideband and running 32 and 64 kb/s
> in fullband mode (even if the input is a 32 kHz signal).
>
> For the very low delay tests (10 ms frame size), I think all the listed
> rates should be using fullband mode except the 32 kb/s.
>
> That's it for now. Any thoughts?
>
> 	Jean-Marc
>
>
> On 11-04-14 11:16 PM, Jean-Marc Valin wrote:
>> Hi Anisse,
>>
>> I gave some more thought on your proposed test plan and as Cullen
>> suggested, I think the main cause of disagreement is not that much on
>> the testing, but on the conditions for publishing (large number of BT,
>> NWT). Considering that ultimately, the decision to publish a spec is
>> always based on WG consensus, then I think that problem can be
>> completely bypassed. Once we make it up to the individuals to decide,
>> then we can focus on "simply" designing a good test.
>>
>> Overall I thought the conditions you were proposing in section 2 were
>> pretty reasonable. There's a few details like selecting existing rates
>> for codecs like Speex and iLBC, but that should be easy to solve. Once
>> these are sorted out, interested parties (we had several hands raised in
>> the last meeting) can start testing and we then let each individual
>> decide on whether the codec is any good based on the results of the tests.
>>
>> Sounds like a plan?
>>
>> Jean-Marc
>>
>>
>> On 11-04-13 03:32 AM, Anisse Taleb wrote:
>>> Hi,
>>> Please find attached a first draft of a test plan of the IETF codec
>>> (Opus).
>>> The proposal does not claim to be complete, there are still many
>>> missing things, e.g. tandeming cases, tests with delay jitter, dtx
>>> etc. Consider it as a starting point for discussion where everyone is
>>> welcome to contribute in a constructive manner. Further updates are
>>> planned, but let's see first some initial comments.
>>>
>>> The attachment is a pdf version, please let me know if you would like
>>> to see another format and I would be glad to oblige.
>>>
>>> Comments and additions are welcome!
>>>
>>> Kind regards,
>>> /Anisse
>>> (From La Jolla - San Diego).
>>>
>>>
>>>
>>> _______________________________________________
>>> codec mailing list
>>> codec@ietf.org
>>> https://www.ietf.org/mailman/listinfo/codec
>> _______________________________________________
>> codec mailing list
>> codec@ietf.org
>> https://www.ietf.org/mailman/listinfo/codec
>>
>>
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec
>
>