Re: [Nmlrg] Machine Learning in network - solicitation for use cases

["Liubing (Leo)" <[e-mail]>]

Hi Sebastian,

Thanks for your further explanation. 
Please see inline.

> >> On 08/09/2015 16:00, Liubing (Leo) wrote:
> >>
> >> ...
> >>> But I'm curious about what is the item that could be labeled as "This is
> not
> >> an attack " or " You missed an attack ". E.g., the item is an packet, a
> stream,
> >> or any other kind of N-tuple things.
> >>
> >> The two cases are rather different.
> >>
> >> 1. The system signals "attack in progress" to the NOC. The operators have
> a
> >> look and decide that there is no attack, it is just some unusual traffic.
> >> (Example: you are live-streaming the Olympic Games. Two seconds after
> the
> >> end of the 100 metres final, there is an enormous burst of traffic.
> >> The machine learning system signals an attack, because it was not trained
> on
> >> the data set from the previous Olympic Games.)
> >>
> >> In this case the NOC operators urgently tell the algorithm it is wrong.
> >> It needs to learn that the signature of a sudden burst just after the end of
> an
> >> event is less likely to be an attack than a sudden burst at another time.
> >>
> >> 2. Someone invents a new kind of DDoS attack, which is therefore not in
> the
> >> historical training data. The system doesn't identify it.
> >> In this case, the NOC operators tell the algorithm "Attack started at
> <time>."
> >
> > [Bing] It feels like the learning objects are mostly traffic burst events?
> When traffic burst happens, the machine judges whether it's a DDoS or not.
> > Then the training data might be a bunch of traffic burst evens marked as
> normal or abnormal. And the challenge should be how to pick a set of
> features out of a burst event for the machine learning program to discover
> the pattern of normal/abnormal classification.
> >
> > This is just my hypothetical case, could be all wrong.
> 
> As you say, the point really is the representation you choose, i.e. the
> features your system uses.  If you encode traffic as bps and pps, then bursts
> won’t tell you anything.  Both in the case of a flash crowd and during a
> volumetric DDoS attack these features would increase.  You can predict a
> little bit more if you also incorporate bps/pps, i.e. bits-per-packet, as in
> reality DDoS attacks typically cause a shift there, but this is all pretty coarse.

[Bing] Indeed. The trick/art is in feature selection.
However, feature selection is basically made by man who understands both the application and the machine learning well. We were always wondering, is there any possibility that machine can select features by itself dynamically according to some general/universal methods.

> And what might be a DDoS attack for a customer might not be considered an
> attack for the carrier - it may even go unnoticed by the carrier as it is too
> low-volume when compared to its usual backbone traffic levels.   What
> this example should tell: for reliable attack detection choosing features that
> are independent of traffic volume is important.  Looking at volume/bursts
> can only be indicative.

[Bing] Yes, it should be one-sided to consider DDoS as "burst" events, especially in a carrier aspect.

> >> This automatically becomes high quality training data for the algorithm:
> the
> >> signature of the new traffic at that time is 100% certain to be an attack.
> >
> >> I think the hard part is extracting useful signatures from the traffic stream
> in
> >> real time; the learning/training part is fairly standard.
> > [Bing] When you said the signature of "100% certain", my perception is
> that it is something like the typical virus detection approach, which doing
> exact match of a particular piece of code that could identify a virus.
> > If my perception was correct, I think the signatures are some special
> combinations of the features (as mentioned above) where the classification
> pattern has a 100% confidence. Then I think it doesn't need to extract the
> signatures in real time, because the features are all pre-defined.
> 
> In both examples the difficulty is backwards-mapping from features to
> packets/flows/etc.  Typically, your features are derived from a set of
> packets packets/flows/etc. collected over a specific period of time. 

[Bing] A clarification question: when you said "your features are derived from a set of packets packets/flows/etc", for the "features", you were referring features that selected manually at the system design stage; not dynamically/automatically selected by the machine at the running stage, right?
 
> So, you have to bin the packets/flows/etc. in a sensible way (i.e., not only time)
> before actually extracting feature vectors in order to be able to draw
> conclusions on very specific packets/flows - which is especially required for
> automatic response.

> BTW: the same applies to the „high quality training data“ as mentioned by
> Brian.  While I agree that manually labelled training data is the gold
> standard, in the case sketched here an operator would only see the
> prediction your ADS makes on the feature vectors, which may be based on a
> bunch of packets/flows/… which might contain both, legitimate and
> illegitimate activity.  So, being able to map labels down to packets or flows
> will be a time consuming manual process.

[Bing] If I understand you correctly, you basically meant the DDoS detection needs to be done at a integral view level (e.g. a bunch of flows/packets in a time period); but the action needs to be performed on a partial objects (e.g. some specific flows). This really is a puzzle.

However, if the goal of the machine is just alerting the DDoS event, then human only need to label the packet/flow "bin" you mentioned, and save a large number of examples of the labeled "bin", then it might be much easier for labeling. But of course, such kind of a system that can only "alert" DDoS makes much less sense than a system that can action on specific packets/flows.

Best regards,
Bing

> sebastian