Re: [bmwg] draft-green-bmwg-seceff-bench-meth-00

[Kenneth Green <[e-mail]>]

Dennis,

Thank you for your comments and my apologies for the delayed response, I never saw this message in my email but fortunately a colleague recently forwarded a copy to me.

Selecting attacks 
=============
This is indeed a pivotal issue and probably the most difficult to resolve.  To a degree the same issue challenges the security performance ID both in terms of the attacks and the background traffic mix. 

>From a purist benchmark perspective a statically defined set of attacks is the best way to ensure apples-to-apples comparisons between test results. However, the very nature of this security effectiveness testing requires the set of attacks to be appropriately current or else there will be limited value to the measured outcome in terms of relevance to operation in the contemporary threat environment.

Even if we cannot be totally prescriptive about the list of attacks, the document will define how the measurements are made and how traffic and results are to reported and this will underpin the validity of comparisons between test runs.

Addressing the problem of attack list definition is work-in-progress and it is early days.

Mechanisms for distinguishing between legal and illegal traffic
================================================
My premise is that it is central to the role of all defensive devices that they must have one or more mechanisms to distinguish between legal and illegal traffic. You correctly point out that there are a variety of mechanisms (signature/pattern recognition, reputation etc.) but that does not change the fact that discrimination is the task at hand.

It is possible that not all features or all classes of defensive box will fall under the umbrella of this draft. The goal is to define tests that are broadly applicable.

Evasions
=======
Yes, evasions are on the list of aspects to cover as we write the scenarios.

Not evil but illegal
==============
After reviewing the definitions in RFC2647 my conclusion at this time is that "Illegal" is the appropriate term since it can encompass the concepts of both malicious traffic and banned traffic.

Regards,
Kenneth

Kenneth Green
Solution Architect
Ixia

From: [e-mail] [mailto:[e-mail]] On Behalf Of Dennis Cox
Sent: Saturday, October 29, 2011 5:50 AM
To: Kenneth Green; [e-mail]
Subject: Re: [bmwg] draft-green-bmwg-seceff-bench-meth-00

Kenneth, 

 I read the draft you provided. Some questions and suggestions since you are soliciting for them.

How will you select the attacks? Will you leverage CVEs? Or is this draft meant to be away of measuring two different flows? The reason I ask is that one companies data leakage is another companies standard traffic. You state that "all of these defensive solutions have in common ... distinguish between evil ... and good traffic". With the list such as anti-virus and anti-spam to data leakage listed, I would have to disagree. Solutions such as these at the eye of the beholder and each users requirements and each technologies inspection is quite different. 

One example: Anti-Spam companies use technologies from reputation based (Cisco IronPort) to RBL. 

How about evasions? Quite a large number of evasion techniques are used by people how attack networks. Will the draft take into account multiple language evasions? Or fragmentation and reassembly evasions? Will you rate the effectiveness of the device in handling attacks each type of evasion?

In order to test security effectiveness it comparing between "good" traffic and "evil" traffic, I would think that whatever application protocol the "evil" traffic uses the good traffic would need to use as well. That should make section 3.1 easy to determine. 

Also, evil may not be the best word, perhaps malicious might be a bit used instead.

Dennis