Re: [rtcweb] Agenda time request for draft-dbenham-webrtcvideomti

[Randell Jesup <[e-mail]>]

On 2/26/2013 3:29 PM, Adam Roach wrote:
> I want to caution people about what they are -- and are not -- seeing 
> in the demonstration videos pointed to by this draft. I don't think 
> the draft is intentionally deceptive on this point, but a casual 
> reader might fail to make a somewhat subtle (but catastrophically 
> important) distinction.
>
> The videos in section 4 are a pretty convincing comparison between 
> hardware encoding and software encoding on same-class platforms. They 
> happen to use different codecs, but that's a non-factor compared to 
> the overwhelming difference between encoding horsepower.

Not only that, but you're comparing VERY different applications with no 
synchronization of settings, of network bandwidth, etc.  Bria/VP8 vs 
Facetime/H.264-more-than-baseline-and-not-even-proposed-here? Not a 
useful comparison.  Similar arguments for all the demos (I won't call 
them tests).

I looked at these originally.  I've worked with Bria in the past. I'm 
certain the bandwidth usage was different.  I'm sure the bandwidth 
adaptation is different.  Application logic is different. Delay almost 
certainly has nothing to do with the codec, but is largely the result of 
jitter buffers/avsync and other issues in the app/browser/os.  Etc, etc.

You can't even begin a comparison of *codecs* without normalizing 
settings *and* measuring how much they actually use, loss, etc.  For 
another example (on the other side of this argument), see the videos 
posted by Google during the last IETF - at the low end of bandwidth 
settings (and this actually is a controlled test with settings), VP8 
looks FAR better.  Until I looked closer, and asked some questions: it 
turned out that while both were *set* for 100Kbps (IIRC), the VP8 
bitrate controller decided it would look too bad or that the bitrate was 
too low for that res/framerate, and ran closer to 200K.

>
> What they absolutely don't show is a comparison of H.264 to VP8. I 
> don't beleive the authors intend to do so, but other parties might be 
> tempted to construe the videos in that light. Doing so would be a 
> straightforward application of "trick 3b" from "How to cheat on video 
> encoder comparisons" (http://x264dev.multimedia.cx/archives/472). Of 
> *course* coding on a DSP will look better than coding on a CPU that 
> draws roughly the same power.
>
> I'm not discounting the availability of H.264 hardware encoders as a 
> factor, and the videos do a pretty good job of showing off the 
> advantages of hardware encoding. But that's all they show.

And I'm not even sure they show the advantage of hardware encoding.

*If* we're going to discuss codec quality, we need (much) better data 
than this.  Google's comparisons from the last IETF are much more 
scientific (though with the caveat I mentioned you have to watch out 
for).  I'll also assert that if the difference is minor (+-10 or 20% 
either way - number pulled out of air) it's irrelevant for this 
discussion.  Even major differences would require discussion as to 
whether they matter.  I personally don't see a need to discuss it, 
unless someone has hard facts that one or the other falls down badly by 
comparison - and that we don't believe it's something that can be fixed 
(especially if it's an implementation flaw).

-- 
Randell Jesup
[e-mail]