Re: Comments on bmwg-secperf-04

[David Newman <[e-mail]>]

Hi Kevin,

Thanks very much for the feedback. Insightful comments, as usual.

Comments inline:

> In section 3.4, Bit Forwarding Rate, you write:  "...this
> measurement counts bits per second rather than frames per
> second.  Per-frame metrics are not meaningful in the context
> of a flow of application data between endpoints."
> 
> Two comments.  First, if you leave the above declaration (i.e.,
> per frame metrics are not meaningful), you may wish to offer a
> explanation as to your rationale. This would help the reader
> build an understanding as to your conclusion.
> 
> Second, if per-frame performance is meaningless in the firewall
> context, (I'm not sure I _totally_ agree), why are you carrying the
> frame overhead in the metric versus some sort of Effective Bit
> Forwarding Rate (bit fwd rate of data payload component only?)
> 
Yes, this text is problematic.

An earlier implementation of our testing tool did exactly what you
suggest--it measured payload only, as reassembled at the application
layer. It was with reassembly in mind that I proposed a metric in bits
or bytes rather than frames per second. We no longer test that way; I'm
much more interested in activity on the wire, since doing so mitigates
the effects of the testing platform's upper-layer software, OS, drivers,
etc. 

Still, we're dealing with applications here and not streams of same-size
packets. Sniffing any ftp or http session will reveal packet sizes all
over the map (and acknowledgments, if we're using half-duplex links).
That makes per-frame metrics difficult. What frame size am I measuring?
Should I average all the frame sizes I see in a given capture? Use a
median? A harmonic mean? 

Bit-based metrics don't raise those questions. I can always measure how
much data I moved per unit of time, regardless of how it got chopped up
along the way.

In terms of fixing this, I'd suggest replacing the last sentence quoted
with:

"Per-frame metrics are not meaningful in the context of a flow of
application data, where data may be segmented into frames of variable
size."

> 
> Section 3.7.  Connection.  I would move this wonderfully defined
> abstration ahead of the connection-related metrics (e.g. concurrent
> connections.)  I had lots of questions about section 3.6, concurrent
> connections, until I read your definition of connection... ;-)
> 

This is just a function of sorting the definitions in alphabetical
order. At the Munich meeting Jim McQuaid noted that the alpha sorting of
RFC 1242 allowed for very fast lookups of any term. That's why I used
it. Not sure there are enough metrics defined in this ID to justify
sorting by type of metric.

> 
> Section 1. Introduction.  "The primary metrics used in this document
> are bit forwarding rate and connections."  A nit.  As you defined
> in section 3.6, a connection is a abstraction, not a metric. Perhaps,
> "... a bit forwarding rate metric and connection-related metrics."

Good catch. That should be corrected.

dn