Re: [codec] Compare tool sensitivity

[Jean-Marc Valin <[e-mail]>]

Hi Erik,

Sorry for taking so long to commit this change to the repository. The
new tool is now in the Opus master branch. Koen and I have come up with
a better comparison tool that is based on a spectral distance estimate
(derived from the Itakura-Saito distance) rather than on the spectrum of
the error signal. That new tool is better at throwing out irrelevant
information (such as small phase mis-alignments) and is also much more
permissive than the previous tool was. The tool maps the spectral
distance to an (arbitrary) quality scale that goes from 0% to 100%,
where 0% is the passing threshold (negative means the comparison fails).

Using the last comparison tool, we had a few cases where the fixed and
float were considered too far to pass. With the new tool, the worst
score for any of the -10 test vectors is now 89%. The passing threshold
(0%) is set to a level that's equivalent to white noise with 48 dB SNR
-- about what you get from a cassette tape. I think this should allow
quite a bit of freedom in decoder implementations. This was verified
when both of Tim's independent re-implementations of SILK (on
fixed-point, one float) passed the test.

Let us know if you have any questions/issues. You can get the new tool
from the master branch, or you can use this link just to see the updated
file:
http://git.xiph.org/?p=opus.git;a=blob;f=src/opus_compare.c

Cheers,

	Jean-Marc

On 08/02/12 09:10 AM, Erik Norvell wrote:
> Hi Koen, all
> 
> This issue has been silent for some time. I am wondering if there has been any progress making an agreeable proposal for the conformance testing? 
> 
> Best regards,
> Erik 
> 
> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On Behalf Of Koen Vos
> Sent: den 17 november 2011 03:49
> To: Ron
> Cc: [e-mail]
> Subject: [codec] Compare tool sensitivity
> 
> (Changed the thread subject)
> 
> Ron wrote:
>> There is one requirement.  Good interoperability.
> 
> I agree completely.  All compliant future encoders and decoders should work well together.  This also prevents embrace-and-extend tactics.
> 
> However, within the space of guaranteed interop, there are benefits to being lenient:
> 
> 1. The decoder may be improved for specific use cases, by finding a different tradeoff between quality, delay, complexity, etc.  For instance in a non-real time application it may help to have a resampler with higher quality, at the cost of higher delay and complexity.
> 
> 2. A limited decoder may be better.  For instance, an Opus-to-PSTN gateway only needs a decoder to generate 8 kHz output signals.  Allowing such a decoder enables optimizations in terms of memory and CPU.  (Of course the decoder would still have to decode any Opus bitstream, including one with more than NB content.)  
> 
> 3. The current test disqualifies a huge space of decoders that sound (virtually) identical.  Simply getting the sign wrong of the output signal shouldn't make a company liable for a patent lawsuit.
> 
> A non-goal for the WG, in my opinion, is to ensure that all users get the quality they are deemed to deserve.  While nobody wants decoders that sound like crap, few users will stick with an application that plays out a jingle whenever an Opus bitstream is received.  And we already allow crappy _encoders_.  I prefer the market place to weed out crap over threats of litigation.
> 
> While there's no perfect solution, I think Jean-Marc and I will come up with a compare tool that provides a small -but useful- amount of freedom to change the decoder without breaking the interop guarantee.
> 
> koen.
> 
> 
> ----- Original Message -----
> From: "Ron" <[e-mail]>
> To: [e-mail]
> Sent: Thursday, November 17, 2011 3:18:14 AM
> Subject: Re: [codec] WGLC #2 for draft-ietf-codec-opus-10
> 
> On Wed, Nov 16, 2011 at 11:37:12AM +0100, Christian Hoene wrote:
>> Hi,
>>
>> I would suggest to split two things: 
>>
>> First, we need a notion for standard conformance because of the legal 
>> IPR requirements (can be more relaxing).
>> Second, we need some tests in respect to interoperability (should be 
>> more straight).
>>
>>  Both tests need to fulfill different requirements.
> 
> I don't see these as different requirements at all.
> 
> The IPR in this case is not being used to protect a revenue stream garnered from unwitting users of the technology - it is being used to protect the interoperability guarantees that can be offered to end users.
> 
> Separating these two things, and watering down the protection offered by the IPR does not offer any benefit to users, only to implementers who might be tempted to compromise on interoperability.  Possibly for 'practical'
> reasons, and possibly for 'commercial' reasons of their own - but their reasons don't matter, only the end effect, a degraded user experience, is the important consideration (to avoid) here.
> 
> There is one requirement.  Good interoperability.  IPR grant conformance is the only binding method we have of ensuring this.  As heretical as it may seem to the FRAND profit seekers - the real profit to be made here from patents is an enhanced and assured user experience.
> 
> Of course, since I don't actually own any of the IPR, I can't speak for the people who do, or for their motives - but this is what I see as their most valuable gift to us, above and beyond the actual technology itself.
> 
> 
>> Also, I like Erik's idea to allow for algorithms to replace patented 
>> stuff in the codec - even if the quality is a bit worse.
> 
> I think this is a fallacy.  It was argued that this extra freedom might allow people to produce a 'better' decoder.  That seems like nonsense to me, the decoder is never going to be able to recover information that simply isn't there - the best it can do is recover all of it with tight fidelity.  Which is what the current conformance test tries to measure.
> 
> It was also argued that this freedom might allow people to produce a worse decoder.  That seems like something we should strongly avoid.
> There seem to be only three reasons to do this:
> 
>  - Save cycles on underpowered systems.
>    But the requirements already include assessing complexity guarantees
>    are within reasonable bounds.
> 
>  - Sidestep the IPR to produce systems that are not fully interoperable
>    but can still claim (some semblance of) conformance.
> 
>  - Provide distortive 'enhancements' that some users may assess as being
>    pleasing on a first brief listen, but which reduce the real fidelity
>    of the output.   (eg. simulate mp3 distortion for the people who
>    listening tests actually prove like that more than lossless original)
> 
> Both those latter cases carry a danger of sounding like crap if future
> (real) enhancements are made to the encoder which actually does increase the real fidelity and information content of the encoded signal.
> 
> The likelihood of such improvements being made to the encoder is high, so I think this is a real jeopardy which we should not inflict upon end users or limit future encoder work by.
> 
> The decoder should decode the bitstream accurately.  If it cannot, then it should not claim to decode Opus.  People are always free to sidestep the IPR and make something that sounds worse.  They just shouldn't be encouraged or allowed to then also misrepresent that as Opus and mislead the users that this group set out to serve.  As some of the liaisons have reminded us, we're here because we wanted to *raise* the bar, not to just continue Business As Usual in patented codec proliferation.
> So we should do just that.  And make everyone happy :)
> 
> IMO
> 
>  Ron
> 
> 
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec