Re: [codec] requirements #12 (closed): bit-exact vs. bit-compatible?

[Koen Vos <[e-mail]>]

A similar tool is in the code in silk\test\signalCompare.c.  It's based on LPC, and measures the SNR in a partially-whitened domain.  For music the critical-band approach of Jean-Marc's tool may work better, not sure.

One question is about what implementation should be the reference for compliance testing: floating-point or fixed-point?

A fixed-point reference allows for bit-exact implementations, which is nice.  And fixed-point reference implementations are the norm in speech coding.
However, CELT's floating-point implementation has a larger dynamic range than the fixed-point version, and is thus arguable better.  A fixed-point reference could not test this extra dynamic range.

Any suggestions?
koen.


----- Original Message -----
From: "Jean-Marc Valin" <[e-mail]>
To: "Stephen Botzko" <[e-mail]>
Cc: [e-mail]
Sent: Tuesday, January 25, 2011 8:29:25 AM
Subject: Re: [codec] requirements #12 (closed): bit-exact vs. bit-compatible?

Just to give an idea of what I'm talking about in terms of compliance test, 
here's a tool I just wrote that could be used for testing. There may need 
to be a bit of refining and playing with the thresholds, but the main idea 
is there:

http://jmvalin.ca/misc_stuff/compare.m

It works with Octave, and is likely very easy to get to work with Matlab. 
Any comments?

	Jean-Marc

On 11-01-25 10:57 AM, Jean-Marc Valin wrote:
> Hi Stephen,
>
> The reference implementation includes both floating-point and fixed-point,
> so I'm not sure it's a good idea to say that one is "better" than the
> other. As for IEEE 754-2008, as far as I know it is not a "bit-exact"
> standard either when it comes to some operations (e.g. transcendental
> function) and even for other operations, most compilers I know of are not
> strict IEEE-754 (at least by default) because of issues such as the x86
> extended precision operations.
>
> Regarding MPEG bit-streams, I'm not sure where to get confirmation, but
> this is what Wikipedia has to say about MPEG-1 Layer 3:
>
>
> "Decoding, on the other hand, is carefully defined in the standard. Most
> decoders are "bitstream compliant", which means that the decompressed
> output – that they produce from a given MP3 file – will be the same, within
> a specified degree of rounding tolerance, as the output specified
> mathematically in the ISO/IEC standard document (ISO/IEC 11172-3).
> Therefore, comparison of decoders is usually based on how computationally
> efficient they are (i.e., how much memory or CPU time they use in the
> decoding process)."
>
>
> That implies a standard based on "infinite precision", with a degree of
> tolerance for different implementations.
>
> Cheers,
>
> Jean-Marc
>
> On 11-01-25 10:42 AM, Stephen Botzko wrote:
>> It seems to me that the reference encoder and decoder will be bit exact for
>> a given floating point format, correct? So one could specify IEEE 754-2008
>> when bit exactness is needed in the reference (for instance regression
>> testing), but not require it for compliance.
>>
>> Folks who are modifying the reference encoder or decoder algorithms are "on
>> their own" as far as audio quality is concerned. Though I agree with
>> Stephan that we need to address compliance (when exactly does a ported or
>> otherwise modified encoder/decoder become non-compliant?).
>>
>> Stephen Botzko
>>
>> On Tue, Jan 25, 2011 at 9:34 AM, Roman Shpount <[e-mail]
>> <mailto:[e-mail]>> wrote:
>>
>> One more concern that I have related to bit exactness is codec
>> regression testing. One can produce a non-bit-exact encoder that
>> produces reasonable results for a reference decoder, but if paired with
>> a modified, non-bit-exact decoder will produce significant audio
>> artifacts. Since neither decoder or encoder are bit exact we will need
>> to have a test procedure that will validate that both encoder or
>> decoder will not break any standard compliant and non-bit-exact
>> encoders or decoders. Not really sure how this can be done.
>>
>> I might be wrong, but I think MPEG decoders are bit exact and encoders
>> are not.
>> _____________
>> Roman Shpount
>>
>>
>>
>> On Tue, Jan 25, 2011 at 12:03 AM, Jean-Marc Valin <[e-mail]
>> <mailto:[e-mail]>> wrote:
>>
>> Hi Stephan,
>>
>> I understand your concern and I'd be interested if you have
>> alternative ways of handling the licensing to avoid any issue. It's
>> not like this is a unique situation. As far as I know, most (all?)
>> MPEG codecs have similar non-bit exact definitions. I have also
>> heard that they also require some IPR licensing...
>>
>> In general the issue of bit-exactness has been discussed and so far
>> I don't recall many arguing in favor of a bit-exact definition.
>> Most of the concerns that have been expressed are solved by
>> considering that non-bitexact does not mean you cannot be bit-exact
>> with the reference encoder. It merely means that you don't *have*
>> to. So regardless of how conformance is defined exactly, one always
>> has the option of being bit-exact with the reference
>> implementation, which obviously guarantees compliance.
>>
>> As for language mentioning compliance, I believe it belongs more to
>> the guidelines (it's not a requirement of the codec itself), which
>> includes the following text:
>>
>> 4. To reduce the risk of bias towards certain CPU/DSP architectures,
>> ideally the decoder specification should not require "bit-exact"
>> conformance with the reference implementation. The output of a
>> decoder implementation should only be "close enough" to the
>> output of the reference decoder. A comparison tool should be
>> provided along with the codec to verify objectively that the
>> output of a decoder is likely to be perceptually
>> indistinguishable from that of the reference decoder. However,
>> an implementation may still wish to produce an output that is
>> bit-exact with the reference implementation to simplify the
>> testing procedure.
>>
>>
>>
>> Cheers,
>>
>> Jean-Marc
>>
>>
>> On 11-01-24 11:02 PM, Stephan Wenger wrote:
>>
>> Hi all:
>>
>> Let me speak once more against this decision (if such a
>> decision were
>> really made; see the p.s.).
>>
>> There are currently three IPR disclosures against the codec
>> draft and/or
>> its predecessors. The Xiph disclosure is at this point a
>> placeholder
>> (Xiph folks: it's time to fix that!). However, the two other
>> disclosures
>> on file provide a patent grant only for necessary patent claims
>> and only
>> when the standard is practiced in full compliance. These terms (in
>> various formulations) are quite common.
>>
>> In order to ensure one has a license (or can rely on a non-assert
>> covenant), one has to ensure one meets the conditions set by the
>> rightholder. On stuff such as reciprocity clauses this is
>> simple. On
>> compliance, it's not always easy.
>>
>> The traditional compliance test for a media codec is a
>> stimulus-response
>> test: you feed test vectors into the codec, and you get
>> results. If the
>> results match, you are in compliance, if not, you are not. Simple.
>>
>> Without bit exactness, the compliance criteria have to be defined
>> differently. We can do so, and, indeed, I recall that this has been
>> mentioned as one plan forward. However, I have seen zero
>> activity in this
>> direction, and I have also not seen any language that mentions
>> this in the
>> requirements draft. I think that the subject of compliance
>> tests, at
>> least in its most basic outline, needs to be documented in the
>> requirements draft. The details can be taken care of elsewhere
>> and later,
>> but not too much later. It should be clear that a codec
>> candidate (if
>> there were more than one) needs to have compliance criteria
>> defined before
>> that codec candidate can become an RFC. Without that, the key
>> goal of the
>> WG, a reasonably freely practicable codec, is just not
>> achievable in the
>> current legal environment (which includes, in this case, the IPR
>> disclosures on file).
>>
>> Of course, it would be sooooo much simpler if we would mandate
>> a bit exact
>> decoder... Is it really that restricting to require that?
>>
>> Stephan
>>
>> P.s.: for the IETF procedures newcomers: humms taken at
>> meetings need to
>> be confirmed on a mailing list, and consensus needs to be
>> declared by the
>> chairs. On this subject, I do recall mailing list discussions after
>> Maastricht, but I do not recall that consensus was reached, yet
>> alone
>> declared. (Unfortunately, I currently don't have the time to go
>> through
>> the mailing list archives to verify my recollection; Sorry.)
>>
>>
>>
>> On 1.24.2011 16:45 , "codec issue tracker"<[e-mail]
>> <mailto:[e-mail]>> wrote:
>>
>> #12: bit-exact vs. bit-compatible?
>>
>> Changes (by gmaxwell@Å ):
>>
>> * status: new => closed
>> * resolution: => worksforme
>>
>>
>> Comment:
>>
>> http://www.ietf.org/proceedings/78/minutes/codec.txt
>>
>> "On the topic of bit exact. Consensus was bit exactness is
>> not required."
>>
>> I believe this issue is already closed.
>>
>> --
>> ------------------------------------+-------------------------------------
>> --
>> Reporter: hoene@Å | Owner:
>> Type: enhancement | Status: closed
>> Priority: minor | Milestone:
>> Component: requirements | Version:
>> Severity: Active WG Document | Resolution: worksforme
>> Keywords: |
>> ------------------------------------+-------------------------------------
>> --
>>
>> Ticket
>> URL:<http://trac.tools.ietf.org/wg/codec/trac/ticket/12#comment:1>
>> codec<http://tools.ietf.org/codec/>
>>
>> _______________________________________________
>> codec mailing list
>> [e-mail] <mailto:[e-mail]>
>> https://www.ietf.org/mailman/listinfo/codec
>>
>>
>>
>> _______________________________________________
>> codec mailing list
>> [e-mail] <mailto:[e-mail]>
>> https://www.ietf.org/mailman/listinfo/codec
>>
>>
>> _______________________________________________
>> codec mailing list
>> [e-mail] <mailto:[e-mail]>
>> https://www.ietf.org/mailman/listinfo/codec
>>
>>
>>
>> _______________________________________________
>> codec mailing list
>> [e-mail] <mailto:[e-mail]>
>> https://www.ietf.org/mailman/listinfo/codec
>>
>>
>>
>>
>> _______________________________________________
>> codec mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/codec
>
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec

_______________________________________________
codec mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/codec