Re: [codec] draft test and processing plan for the IETF Codec

I would also suggest replacing all BT (better than) requirements by NWT (no worse than).

My reasoning is that:
- The WG never had the goal to be better than other codecs (see charter).
- Proving to be better can be very hard, especially when several codecs are close to transparent.  To show significance in that case you'd need a vast number of listeners, which makes a test more cumbersome to perform.

best,
koen.

----- Original Message -----
From: "Jean-Marc Valin" <jmvalin@jmvalin.ca>
To: "Jean-Marc Valin" <jmvalin@jmvalin.ca>
Cc: codec@ietf.org
Sent: Thursday, April 14, 2011 9:49:58 PM
Subject: Re: [codec] draft test and processing plan for the IETF Codec

So here's some more specific comments on actual bitrates:

1) For narrowband Speex, the rates currently listed are 8, 12, 16 kb/s. 
Those should be changed to 8, 11, 15 kb/s to match the actual Speex 
bitrates.

2) For iLBC, the rates currently listed are 8, 12, 16 kb/s. I think we 
should only use 15.2 kb/s for iLBC. There's another rate, which is 13.33 
kb/s but that's for 30 ms frames so it's not very interesting.

3) For Speex wideband, the rates currently listed are 12, 24, 32 kb/s. I 
think Speex wideband around 12 kb/s is just crap. Worth testing would be 
20.6 and 27.8 kb/s.

4) For super-wideband Speex, I recommend just dumping that. This Speex 
mode was a mistake right from the start and usually has worse quality 
than wideband Speex.

Regarding super-wideband, one thing to keep in mind is that Opus defines 
super-wideband as having a 12 kHz audio bandwidth (24 kHz sampling 
rate). This makes comparisons with other codecs more difficult. The 
rates currently listed for super-wideband are 24, 32, 64 kb/s. I 
recommend running 24 kb/s in super-wideband and running 32 and 64 kb/s 
in fullband mode (even if the input is a 32 kHz signal).

For the very low delay tests (10 ms frame size), I think all the listed 
rates should be using fullband mode except the 32 kb/s.

That's it for now. Any thoughts?

	Jean-Marc

On 11-04-14 11:16 PM, Jean-Marc Valin wrote:
> Hi Anisse,
>
> I gave some more thought on your proposed test plan and as Cullen
> suggested, I think the main cause of disagreement is not that much on
> the testing, but on the conditions for publishing (large number of BT,
> NWT). Considering that ultimately, the decision to publish a spec is
> always based on WG consensus, then I think that problem can be
> completely bypassed. Once we make it up to the individuals to decide,
> then we can focus on "simply" designing a good test.
>
> Overall I thought the conditions you were proposing in section 2 were
> pretty reasonable. There's a few details like selecting existing rates
> for codecs like Speex and iLBC, but that should be easy to solve. Once
> these are sorted out, interested parties (we had several hands raised in
> the last meeting) can start testing and we then let each individual
> decide on whether the codec is any good based on the results of the tests.
>
> Sounds like a plan?
>
> Jean-Marc
>
>
> On 11-04-13 03:32 AM, Anisse Taleb wrote:
>> Hi,
>> Please find attached a first draft of a test plan of the IETF codec
>> (Opus).
>> The proposal does not claim to be complete, there are still many
>> missing things, e.g. tandeming cases, tests with delay jitter, dtx
>> etc. Consider it as a starting point for discussion where everyone is
>> welcome to contribute in a constructive manner. Further updates are
>> planned, but let's see first some initial comments.
>>
>> The attachment is a pdf version, please let me know if you would like
>> to see another format and I would be glad to oblige.
>>
>> Comments and additions are welcome!
>>
>> Kind regards,
>> /Anisse
>> (From La Jolla - San Diego).
>>
>>
>>
>> _______________________________________________
>> codec mailing list
>> codec@ietf.org
>> https://www.ietf.org/mailman/listinfo/codec
> _______________________________________________
> codec mailing list
> codec@ietf.org
> https://www.ietf.org/mailman/listinfo/codec
>
>
_______________________________________________
codec mailing list
codec@ietf.org
https://www.ietf.org/mailman/listinfo/codec