Re: [codec] requirements #12 (closed): bit-exact vs. bit-compatible?

[Roman Shpount <[e-mail]>]

One more concern that I have related to bit exactness is codec regression
testing. One can produce a non-bit-exact encoder that produces reasonable
results for a reference decoder, but if paired with a modified,
non-bit-exact decoder will produce significant audio artifacts. Since
neither decoder or encoder are bit exact we will need to have a test
procedure that will validate that both encoder or decoder will not break any
standard compliant and non-bit-exact encoders or decoders. Not really sure
how this can be done.

I might be wrong, but I think MPEG decoders are bit exact and encoders are
not.
_____________
Roman Shpount


On Tue, Jan 25, 2011 at 12:03 AM, Jean-Marc Valin <[e-mail]>wrote:

> Hi Stephan,
>
> I understand your concern and I'd be interested if you have alternative
> ways of handling the licensing to avoid any issue. It's not like this is a
> unique situation. As far as I know, most (all?) MPEG codecs have similar
> non-bit exact definitions. I have also heard that they also require some IPR
> licensing...
>
> In general the issue of bit-exactness has been discussed and so far I don't
> recall many arguing in favor of a bit-exact definition. Most of the concerns
> that have been expressed are solved by considering that non-bitexact does
> not mean you cannot be bit-exact with the reference encoder. It merely means
> that you don't *have* to. So regardless of how conformance is defined
> exactly, one always has the option of being bit-exact with the reference
> implementation, which obviously guarantees compliance.
>
> As for language mentioning compliance, I believe it belongs more to the
> guidelines (it's not a requirement of the codec itself), which includes the
> following text:
>
>   4.  To reduce the risk of bias towards certain CPU/DSP architectures,
>       ideally the decoder specification should not require "bit-exact"
>       conformance with the reference implementation.  The output of a
>       decoder implementation should only be "close enough" to the
>       output of the reference decoder.  A comparison tool should be
>       provided along with the codec to verify objectively that the
>       output of a decoder is likely to be perceptually
>       indistinguishable from that of the reference decoder.  However,
>       an implementation may still wish to produce an output that is
>       bit-exact with the reference implementation to simplify the
>       testing procedure.
>
>
>
> Cheers,
>
>        Jean-Marc
>
>
> On 11-01-24 11:02 PM, Stephan Wenger wrote:
>
>> Hi all:
>>
>> Let me speak once more against this decision (if such a decision were
>> really made; see the p.s.).
>>
>> There are currently three IPR disclosures against the codec draft and/or
>> its predecessors.  The Xiph disclosure is at this point a placeholder
>> (Xiph folks: it's time to fix that!).  However, the two other disclosures
>> on file provide a patent grant only for necessary patent claims and only
>> when the standard is practiced in full compliance.  These terms (in
>> various formulations) are quite common.
>>
>> In order to ensure one has a license (or can rely on a non-assert
>> covenant), one has to ensure one meets the conditions set by the
>> rightholder.  On stuff such as reciprocity clauses this is simple.  On
>> compliance, it's not always easy.
>>
>> The traditional compliance test for a media codec is a stimulus-response
>> test: you feed test vectors into the codec, and you get results.  If the
>> results match, you are in compliance, if not, you are not.  Simple.
>>
>> Without bit exactness, the compliance criteria have to be defined
>> differently.  We can do so, and, indeed, I recall that this has been
>> mentioned as one plan forward.  However, I have seen zero activity in this
>> direction, and I have also not seen any language that mentions this in the
>> requirements draft.  I think that the subject of compliance tests, at
>> least in its most basic outline, needs to be documented in the
>> requirements draft.  The details can be taken care of elsewhere and later,
>> but not too much later.  It should be clear that a codec candidate (if
>> there were more than one) needs to have compliance criteria defined before
>> that codec candidate can become an RFC.  Without that, the key goal of the
>> WG, a reasonably freely practicable codec, is just not achievable in the
>> current legal environment (which includes, in this case, the IPR
>> disclosures on file).
>>
>> Of course, it would be sooooo much simpler if we would mandate a bit exact
>> decoder...  Is it really that restricting to require that?
>>
>> Stephan
>>
>> P.s.: for the IETF procedures newcomers: humms taken at meetings need to
>> be confirmed on a mailing list, and consensus needs to be declared by the
>> chairs.  On this subject, I do recall mailing list discussions after
>> Maastricht, but I do not recall that consensus was reached, yet alone
>> declared.  (Unfortunately, I currently don't have the time to go through
>> the mailing list archives to verify my recollection; Sorry.)
>>
>>
>>
>> On 1.24.2011 16:45 , "codec issue tracker"<[e-mail]>  wrote:
>>
>>  #12: bit-exact vs. bit-compatible?
>>>
>>> Changes (by gmaxwell@Š):
>>>
>>>  * status:  new =>  closed
>>>  * resolution:  =>  worksforme
>>>
>>>
>>> Comment:
>>>
>>> http://www.ietf.org/proceedings/78/minutes/codec.txt
>>>
>>> "On the topic of bit exact. Consensus was bit exactness is not required."
>>>
>>> I believe this issue is already closed.
>>>
>>> --
>>>
>>> ------------------------------------+-------------------------------------
>>> --
>>> Reporter:  hoene@Š                 |        Owner:
>>>     Type:  enhancement             |       Status:  closed
>>> Priority:  minor                   |    Milestone:
>>> Component:  requirements            |      Version:
>>> Severity:  Active WG Document      |   Resolution:  worksforme
>>> Keywords:                          |
>>>
>>> ------------------------------------+-------------------------------------
>>> --
>>>
>>> Ticket URL:<http://trac.tools.ietf.org/wg/codec/trac/ticket/12#comment:1
>>> >
>>> codec<http://tools.ietf.org/codec/>
>>>
>>> _______________________________________________
>>> codec mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/codec
>>>
>>
>>
>> _______________________________________________
>> codec mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/codec
>>
>>
>>  _______________________________________________
> codec mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/codec
>