Re: [video-codec] Performances, measurement of same (Re: The Can Has Landed)

["Timothy B. Terriberry" <[e-mail]>]

Harald Alvestrand wrote:
> Just as a matter of curiosity, which state of the art performance is it
> that you think VP9 hasn't achieved yet?

I think if we were trying to develop a competitor to HEVC, we could say 
"Ship VP9" and be done. In fact, that is exactly what Firefox did. But I 
am (personally) worried about what comes after HEVC, and I think we have 
an opportunity to get ahead of the curve here like we did with Opus.

> How should we treat that aspect in evaluation?

I have been thinking about this a lot since Mo asked the question a few 
weeks ago, and I am having a hard time coming up with an answer that is 
not "use the normal IETF consensus process".

We've certainly tried to focus on practical algorithms in Daala. 
However, sometimes it is nice to know what the theoretical best we could 
do actually is, so we know how well some of our cheap hacks are performing.

To use a recent example, until the last few weeks, the way we had been 
implementing our lapped transforms, one needed to know the block sizes 
of your neighbors to know what lapping size to apply. That meant we 
couldn't do an exhaustive search of block sizes without making some kind 
of gross simplifications. We restructured things to remove this 
dependency, even though it meant the filtering was theoretically worse. 
That allowed us to write an exhaustive search, which turned out to be 
between 3% and 11% better than the heuristics we had been using to pick 
block sizes. Until we did this, we didn't have a good idea what kind of 
gains were possible, nor how to change our heuristics to produce better 
results. Now we have something we can look at to tell us both.

So having both the heuristic and the exhaustive search is the ideal 
situation, and sometimes asking for it is reasonable. Sometimes it may 
be more work than people are willing to do up-front to make a decision 
about what is a good idea.

But I will also give a second example, that is even more recent. One 
issue Thomas Davies from Cisco pointed out to us was some kind of noise 
buildup along block boundaries in one of their test clips at low rates, 
something we had not seen in our own testing. We were able to track down 
the problem to certain rounding operations in the pre/post filters and 
in converting from the 12-bit data we use in our transforms down to 
8-bit data we store in our reference frames. Storing 12-bit reference 
frames makes the problem go away, but increases the memory bandwidth 
required by the codec by 50%. But we've observed gains between 0.2 dB 
and (on specially constructed examples) 2 dB by doing so. Is that a cost 
people are willing to pay? I don't know the answer to that question, but 
I do know that the easiest way to measure that, x86 cycle counts, is 
also probably the least informative. It also matters what other 
solutions there are (we have at least one more, with a different set of 
unappealing costs), or how hard we want to look for them.

I have a real hard time seeing how to design an up-front process that 
answers questions like this. Maybe others here have better ideas than I do.