Re: [codec] draft test and processing plan for the IETF Codec

[Koen Vos <[e-mail]>]

Jean-Marc Valin wrote:
> This clarifies a few things, though it makes other aspects more puzzling.

Anisse, I too am confused by your proposed "test plan".

By my count, only 8 of the 32 codec/bitrate requirements have a match in the 
codec requirements document.  Of the other 24, many seem rather contentious.  
Had you wanted to add requirements, you could have suggested and motivated 
them directly to the mailing list or at the meetings a long time ago.

But my problem with your 24 "new" requirements is not just procedural.  
If you read the charter of this WG, you'll see that "beating most existing 
codecs" was never a goal.  The strongest wording about performance is that 
the codec shall enable "high-quality audio services on the Internet".  
Instead of spending countless hours on testing against arbitrary 
requirements, what better way to verify that we've reached that goal than 
deploying it in actual Internet applications?

Early adopters will realize there is always a risk that the bitstream will 
have to change, no matter how much we test.  They'll also know that the 
likelihood of such changes decreases fairly quickly with time and adoption.  

Stephen Botzko recently pointed out that codecs implemented in hardware are 
difficult to upgrade(*).  While true, it's no argument against deploying
sooner rather than later, because no sensible hardware manufacturer is going 
to put OPUS in hardware before it has proven itself in the market place.

The testing done to date by developers and independent parties shows we do 
indeed have "running code."  But the proof of the pudding is in the eating, 
and voluntary deployment in real-world (software) applications is the right 
next step.

best,
koen.

*) http://www.ietf.org/mail-archive/web/codec/current/msg02352.html


----- Original Message -----
From: "Jean-Marc Valin" <[e-mail]>
To: "Anisse Taleb" <[e-mail]>
Cc: [e-mail]
Sent: Wednesday, April 13, 2011 7:04:08 PM
Subject: Re: [codec] draft test and processing plan for the IETF Codec

Hi Anisse,

Thanks for the answers. This clarifies a few things, though it makes 
other aspects more puzzling. From what I understand now, what you're 
proposing is that before Opus can be published, we need to verify that 
Opus 8 kb/s be better than Speex at 8 kb/s, no worse than iLBC, that 
Opus 12 kb/s be better than Speex at 12 kb/s, no worse than ilbc, no 
worse than AMR-NB at 12.2 kb/s, no worse than EVRC-NW at 8.3 kb/s, that 
Opus WB 12 kb/s be better than Speex at 12 kb/s, better than G.722 at 56 
kb/s, and so on.

In total, I count 27 codecs/bitrates combinations (excluding GSM-FR) we 
must be better than or no worse than. Considering the three proposed 
packet loss rates and two rate management options (CBR/VBR), we get a 
total of 162 conditions (75 BT and 87 NWT) that need to be met, 
involving 11 reference codecs. As a comparison, I looked at the example 
test plans that Paul sent. One has 4 reference codecs and the other has 
3. It's not clear from those documents how many BT and NWT comparisons 
are involved.

Cheers,

    Jean-Marc

On 11-04-13 07:56 PM, Anisse Taleb wrote:
> Dear Jean Marc.
>
> Thank you for your valuable comments.
>
>> 1) How much is "very low delay" and how much is "medium delay"?
>
> This is based on section [5.1. Operating space] of the requirements document http://www.ietf.org/id/draft-ietf-codec-requirements-02.txt.
> "medium delay" (20-30ms), while a few require a "very low delay" (<  10 ms). Would you like to contribute some precise delay operating points for Opus?
>
>> 2) What do "NWT" and "BT" mean?
> Sorry for this, I should have defined these abbreviations,
>
> NWT = Not Worse Than.
> BT = Better Than.
>
>> 3) What's the difference between the "Reference Codecs" column and the
>> "Requirements" column?
>
> I am not entirely happy with how this table looks like. Essentially, the first column is the collection of codecs, while the requirements column details the actual requirements. I will group these in a next version.
>
>> 4) How do you define the rates that do not exist in some codecs? For
>> example, I see a reference to 12 and 16 kb/s for Speex, which has no such
>> rates, and to 8, 12, 16 kb/s for iLBC, which does not have these rates
>> either.
>
> Thank you for pointing this out. If the rates are not supported, I would suggest to use closest bitrates.
> In this initial version, I didn't check if all codecs could be operated at the bitrates being proposed.
> Would you like to suggest on which rates ilbc and speex operated?
>
> Kind regards,
> /Anisse
>
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec
>
>
_______________________________________________
codec mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/codec