Re: [Nmlrg] Machine Learning in network - solicitation for use cases

[Sebastian Abt <[e-mail]>]

> Am 08.09.2015 um 10:13 schrieb Liubing (Leo) <[e-mail]>om>:
> 
> Hi Brian,
> 
> Thanks for your elaborating explanation.
> Please see inline.
> 
>> -----Original Message-----
>> From: Brian E Carpenter [mailto:[e-mail]]
>> Sent: Tuesday, September 08, 2015 12:39 PM
>> To: Liubing (Leo); Sheng Jiang; Dacheng Zhang; [e-mail]
>> Subject: Re: [Nmlrg] Machine Learning in network - solicitation for use cases
>> 
>> On 08/09/2015 16:00, Liubing (Leo) wrote:
>> 
>> ...
>>> But I'm curious about what is the item that could be labeled as "This is not
>> an attack " or " You missed an attack ". E.g., the item is an packet, a stream,
>> or any other kind of N-tuple things.
>> 
>> The two cases are rather different.
>> 
>> 1. The system signals "attack in progress" to the NOC. The operators have a
>> look and decide that there is no attack, it is just some unusual traffic.
>> (Example: you are live-streaming the Olympic Games. Two seconds after the
>> end of the 100 metres final, there is an enormous burst of traffic.
>> The machine learning system signals an attack, because it was not trained on
>> the data set from the previous Olympic Games.)
>> 
>> In this case the NOC operators urgently tell the algorithm it is wrong.
>> It needs to learn that the signature of a sudden burst just after the end of an
>> event is less likely to be an attack than a sudden burst at another time.
>> 
>> 2. Someone invents a new kind of DDoS attack, which is therefore not in the
>> historical training data. The system doesn't identify it.
>> In this case, the NOC operators tell the algorithm "Attack started at <time>."
> 
> [Bing] It feels like the learning objects are mostly traffic burst events? When traffic burst happens, the machine judges whether it's a DDoS or not.
> Then the training data might be a bunch of traffic burst evens marked as normal or abnormal. And the challenge should be how to pick a set of features out of a burst event for the machine learning program to discover the pattern of normal/abnormal classification.
> 
> This is just my hypothetical case, could be all wrong.

As you say, the point really is the representation you choose, i.e. the features your system uses.  If you encode traffic as bps and pps, then bursts won’t tell you anything.  Both in the case of a flash crowd and during a volumetric DDoS attack these features would increase.  You can predict a little bit more if you also incorporate bps/pps, i.e. bits-per-packet, as in reality DDoS attacks typically cause a shift there, but this is all pretty coarse.  And what might be a DDoS attack for a customer might not be considered an attack for the carrier - it may even go unnoticed by the carrier as it is too low-volume when compared to its usual backbone traffic levels.   What this example should tell: for reliable attack detection choosing features that are independent of traffic volume is important.  Looking at volume/bursts can only be indicative. 

>> This automatically becomes high quality training data for the algorithm: the
>> signature of the new traffic at that time is 100% certain to be an attack.
> 
>> I think the hard part is extracting useful signatures from the traffic stream in
>> real time; the learning/training part is fairly standard.
> [Bing] When you said the signature of "100% certain", my perception is that it is something like the typical virus detection approach, which doing exact match of a particular piece of code that could identify a virus.
> If my perception was correct, I think the signatures are some special combinations of the features (as mentioned above) where the classification pattern has a 100% confidence. Then I think it doesn't need to extract the signatures in real time, because the features are all pre-defined. 

In both examples the difficulty is backwards-mapping from features to packets/flows/etc.  Typically, your features are derived from a set of packets packets/flows/etc. collected over a specific period of time.  So, you have to bin the packets/flows/etc. in a sensible way (i.e., not only time) before actually extracting feature vectors in order to be able to draw conclusions on very specific packets/flows - which is especially required for automatic response.

BTW: the same applies to the „high quality training data“ as mentioned by Brian.  While I agree that manually labelled training data is the gold standard, in the case sketched here an operator would only see the prediction your ADS makes on the feature vectors, which may be based on a bunch of packets/flows/… which might contain both, legitimate and illegitimate activity.  So, being able to map labels down to packets or flows will be a time consuming manual process.

sebastian