[codec] Non-bitstream impacting encoder changes and testing

[Gregory Maxwell <[e-mail]>]

I have some fairly minor patches queued up for which improve some
of the encoder signal analysis and should produce some minor quality
improvements.  I have more development testing to do before I consider
them done, but I've been dragging my feet because I'm not quite sure
how this sort of change will interact with testing.

In Opus the encoder behavior is non-normative— precisely so the
format could continue to enjoy encoder-side improvements (like we've
seen with Speex, Vorbis, Theora, and MPEG codecs). But I'm concerned
that some people might consider the testing which has already been
performed to be invalidated by this sort of improvement.

Rather than fall into a pit over this after the fact, I thought it would
be better to ask if WG participants have any views on this subject?

My view is that changes which do not change the bitstream/decoder should
not be considered to invalidate any testing. Even if a change is later
discovered to degrade the performance of the codec, an implementer is
free to use an older encoder, manually back out that change, etc. So,
basically, every test is at least evidence of what can be achieved with
the format even if the encoder is changed later. Good testing practice
should identify the versions used in any case.

The obvious alternatives I see are to simply not improve the encoder
right now, or to work on a separate encoder implementation (a fork of
the reference encoder). I don't either of these are especially good.