Re: [bmwg] Scope of draft-constantine-bmwg-traffic-management-02

note that it is one thing to report usefully on how well a system that performs "strict policing" to a specific data rate 
than it is to report usefully on the operation of a system that does active queue management (e.g. RFC 2309 or
draft-nichols-tsvwg-codel-01.txt) - a simple token bucket device would be quite complex to design a test for - so 
careful scoping is needed

but the underlying question remains - is BMWG in the conformance or the performance space?

Scott

On Sep 22, 2013, at 9:35 AM, "MORTON, ALFRED C (AL)" <acmorton@att.com> wrote:

> Thanks for introducing this topic on the list, Barry.
>  
> There are functions which must perform correctly is all devices
> we've benchmarked, but that has not been the emphasis of the benchmarks.
>  
> you wrote:
> But again, the lack of an objective way to compare these traffic management functions is a gaping hole in the industry
> *and* the importance/interest in proper traffic management is at the fore front of many carriers / providers.
>  
>  
> Say we have configured traffic shapers to perform a specification,
> such as strict policing of <=20Mbps.  Maybe a test of transmission rate
> accuracy would be appropriate here (in addition to the tests Barry mentioned **).
> For example, a device that operated with 18 such policers with 99.9% accuracy,
> but 2 policers with flow rate <95% of the configured value has reached
> some form of capacity limit we can report.
>  
> Al
> (as a participant)
>  
> From: bmwg-bounces@ietf.org [mailto:bmwg-bounces@ietf.org] On Behalf Of Barry Constantine
> Sent: Saturday, September 21, 2013 4:46 PM
> To: bmwg@ietf.org
> Subject: [bmwg] Scope of draft-constantine-bmwg-traffic-management-02
>  
> Hi BMWG,
>  
> Wanted to bring a topic onto the list from IETF 87 and this had to do with the concern that the traffic management benchmarking work was introducing aspects of conformance testing into the framework.
>  
> A summary of the discussion is provided below and we would like to get feedback and drive to closure with the working group.
>  
> To Scott Bradner’s concern with the work:  “function verification sounds a lot like "conformance" and conformance is clearly out of scope for BMWG”.  The specific context Scott referenced was that of a traffic shaper.
>  
> Here are some key paragraphs that we added to -02 to clarify the scope:
>  
> *** Section 1.2
> "   The tests are divided into individual tests and rated capacity tests.
>    The individual tests are intended to verify the traffic management
>    function according to the specifications.  As an example, suppose a
>    traffic shaper is to be tested at a CIR of 20 Mbps.  The intent of
>    the individual test is to test one instance of the shaper and it's
>    ability to properly shape according to the metrics** defined in
>    section 4."
> ** These metrics include burst size achieved, packet loss / sequence,  jitter
>  
> *** Section 3
> "   Also, it is not within scope to perform conformance testing. Tests
>    defined in this framework benchmark the traffic management functions
>    according to the metrics defined in section 4 and do not address any
>    conformance to standards related to traffic management.  Traffic
>    management specifications largely do not exist and this is a prime
>    driver for this framework; to provide an objective means to compare
>    vendor traffic management functions."
>  
> From the co-author’s perspective, what we see in industry today is that vendors advertise functions such as policers, shapers, etc and there is no means to compare performance between them.
>  
> We definitely want to avoid this work being any form of conformance test and realize we must carefully specify the benchmark measurements so as not to cross that line. 
>  
> But again, the lack of an objective way to compare these traffic management functions is a gaping hole in the industry *and* the importance/interest in proper traffic management is at the fore front of many carriers / providers.
>  
> So we’re looking to the folks in BMWG to chime in, help sort out the scope, provide expertise in meaningful benchmark metrics, etc.
>  
> Thank you,
> Barry Constantine
>  
> _______________________________________________
> bmwg mailing list
> bmwg@ietf.org
> https://www.ietf.org/mailman/listinfo/bmwg

Scott O. Bradner
Senior Technical Consultant

Harvard University Information Technology
Innovation & Architecture
+1 617 495 3864
1033 Mass Ave., Room 462
Cambridge, MA 02138
www.harvard.edu/huit