Re: [codec] requirements #12 (closed): bit-exact vs. bit-compatible?

[Jean-Marc Valin <[e-mail]>]

Hi Stephen,

The reference implementation includes both floating-point and fixed-point, 
so I'm not sure it's a good idea to say that one is "better" than the 
other. As for IEEE 754-2008, as far as I know it is not a "bit-exact" 
standard either when it comes to some operations (e.g. transcendental 
function) and even for other operations, most compilers I know of are not 
strict IEEE-754 (at least by default) because of issues such as the x86 
extended precision operations.

Regarding MPEG bit-streams, I'm not sure where to get confirmation, but 
this is what Wikipedia has to say about MPEG-1 Layer 3:


"Decoding, on the other hand, is carefully defined in the standard. Most 
decoders are "bitstream compliant", which means that the decompressed 
output – that they produce from a given MP3 file – will be the same, within 
a specified degree of rounding tolerance, as the output specified 
mathematically in the ISO/IEC standard document (ISO/IEC 11172-3). 
Therefore, comparison of decoders is usually based on how computationally 
efficient they are (i.e., how much memory or CPU time they use in the 
decoding process)."


That implies a standard based on "infinite precision", with a degree of 
tolerance for different implementations.

Cheers,

	Jean-Marc

On 11-01-25 10:42 AM, Stephen Botzko wrote:
> It seems to me that the reference encoder and decoder will be bit exact for
> a given floating point format, correct? So one could specify IEEE 754-2008
> when bit exactness is needed in the reference (for instance regression
> testing), but not require it for compliance.
>
> Folks who are modifying the reference encoder or decoder algorithms are "on
> their own" as far as audio quality is concerned. Though I agree with
> Stephan that we need to address compliance (when exactly does a ported or
> otherwise modified encoder/decoder become non-compliant?).
>
> Stephen Botzko
>
> On Tue, Jan 25, 2011 at 9:34 AM, Roman Shpount <[e-mail]
> <mailto:[e-mail]>> wrote:
>
>     One more concern that I have related to bit exactness is codec
>     regression testing. One can produce a non-bit-exact encoder that
>     produces reasonable results for a reference decoder, but if paired with
>     a modified, non-bit-exact decoder will produce significant audio
>     artifacts. Since neither decoder or encoder are bit exact we will need
>     to have a test procedure that will validate that both encoder or
>     decoder will not break any standard compliant and non-bit-exact
>     encoders or decoders. Not really sure how this can be done.
>
>     I might be wrong, but I think MPEG decoders are bit exact and encoders
>     are not.
>     _____________
>     Roman Shpount
>
>
>
>     On Tue, Jan 25, 2011 at 12:03 AM, Jean-Marc Valin <[e-mail]
>     <mailto:[e-mail]>> wrote:
>
>         Hi Stephan,
>
>         I understand your concern and I'd be interested if you have
>         alternative ways of handling the licensing to avoid any issue. It's
>         not like this is a unique situation. As far as I know, most (all?)
>         MPEG codecs have similar non-bit exact definitions. I have also
>         heard that they also require some IPR licensing...
>
>         In general the issue of bit-exactness has been discussed and so far
>         I don't recall many arguing in favor of a bit-exact definition.
>         Most of the concerns that have been expressed are solved by
>         considering that non-bitexact does not mean you cannot be bit-exact
>         with the reference encoder. It merely means that you don't *have*
>         to. So regardless of how conformance is defined exactly, one always
>         has the option of being bit-exact with the reference
>         implementation, which obviously guarantees compliance.
>
>         As for language mentioning compliance, I believe it belongs more to
>         the guidelines (it's not a requirement of the codec itself), which
>         includes the following text:
>
>         4. To reduce the risk of bias towards certain CPU/DSP architectures,
>         ideally the decoder specification should not require "bit-exact"
>         conformance with the reference implementation. The output of a
>         decoder implementation should only be "close enough" to the
>         output of the reference decoder. A comparison tool should be
>         provided along with the codec to verify objectively that the
>         output of a decoder is likely to be perceptually
>         indistinguishable from that of the reference decoder. However,
>         an implementation may still wish to produce an output that is
>         bit-exact with the reference implementation to simplify the
>         testing procedure.
>
>
>
>         Cheers,
>
>         Jean-Marc
>
>
>         On 11-01-24 11:02 PM, Stephan Wenger wrote:
>
>             Hi all:
>
>             Let me speak once more against this decision (if such a
>             decision were
>             really made; see the p.s.).
>
>             There are currently three IPR disclosures against the codec
>             draft and/or
>             its predecessors. The Xiph disclosure is at this point a
>             placeholder
>             (Xiph folks: it's time to fix that!). However, the two other
>             disclosures
>             on file provide a patent grant only for necessary patent claims
>             and only
>             when the standard is practiced in full compliance. These terms (in
>             various formulations) are quite common.
>
>             In order to ensure one has a license (or can rely on a non-assert
>             covenant), one has to ensure one meets the conditions set by the
>             rightholder. On stuff such as reciprocity clauses this is
>             simple. On
>             compliance, it's not always easy.
>
>             The traditional compliance test for a media codec is a
>             stimulus-response
>             test: you feed test vectors into the codec, and you get
>             results. If the
>             results match, you are in compliance, if not, you are not. Simple.
>
>             Without bit exactness, the compliance criteria have to be defined
>             differently. We can do so, and, indeed, I recall that this has been
>             mentioned as one plan forward. However, I have seen zero
>             activity in this
>             direction, and I have also not seen any language that mentions
>             this in the
>             requirements draft. I think that the subject of compliance
>             tests, at
>             least in its most basic outline, needs to be documented in the
>             requirements draft. The details can be taken care of elsewhere
>             and later,
>             but not too much later. It should be clear that a codec
>             candidate (if
>             there were more than one) needs to have compliance criteria
>             defined before
>             that codec candidate can become an RFC. Without that, the key
>             goal of the
>             WG, a reasonably freely practicable codec, is just not
>             achievable in the
>             current legal environment (which includes, in this case, the IPR
>             disclosures on file).
>
>             Of course, it would be sooooo much simpler if we would mandate
>             a bit exact
>             decoder... Is it really that restricting to require that?
>
>             Stephan
>
>             P.s.: for the IETF procedures newcomers: humms taken at
>             meetings need to
>             be confirmed on a mailing list, and consensus needs to be
>             declared by the
>             chairs. On this subject, I do recall mailing list discussions after
>             Maastricht, but I do not recall that consensus was reached, yet
>             alone
>             declared. (Unfortunately, I currently don't have the time to go
>             through
>             the mailing list archives to verify my recollection; Sorry.)
>
>
>
>             On 1.24.2011 16:45 , "codec issue tracker"<[e-mail]
>             <mailto:[e-mail]>> wrote:
>
>                 #12: bit-exact vs. bit-compatible?
>
>                 Changes (by gmaxwell@Å ):
>
>                 * status: new => closed
>                 * resolution: => worksforme
>
>
>                 Comment:
>
>                 http://www.ietf.org/proceedings/78/minutes/codec.txt
>
>                 "On the topic of bit exact. Consensus was bit exactness is
>                 not required."
>
>                 I believe this issue is already closed.
>
>                 --
>                 ------------------------------------+-------------------------------------
>                 --
>                 Reporter: hoene@Å | Owner:
>                 Type: enhancement | Status: closed
>                 Priority: minor | Milestone:
>                 Component: requirements | Version:
>                 Severity: Active WG Document | Resolution: worksforme
>                 Keywords: |
>                 ------------------------------------+-------------------------------------
>                 --
>
>                 Ticket
>                 URL:<http://trac.tools.ietf.org/wg/codec/trac/ticket/12#comment:1>
>                 codec<http://tools.ietf.org/codec/>
>
>                 _______________________________________________
>                 codec mailing list
>                 [e-mail] <mailto:[e-mail]>
>                 https://www.ietf.org/mailman/listinfo/codec
>
>
>
>             _______________________________________________
>             codec mailing list
>             [e-mail] <mailto:[e-mail]>
>             https://www.ietf.org/mailman/listinfo/codec
>
>
>         _______________________________________________
>         codec mailing list
>         [e-mail] <mailto:[e-mail]>
>         https://www.ietf.org/mailman/listinfo/codec
>
>
>
>     _______________________________________________
>     codec mailing list
>     [e-mail] <mailto:[e-mail]>
>     https://www.ietf.org/mailman/listinfo/codec
>
>
>
>
> _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec