Re: [Nmlrg] Machine Learning in network - solicitation for use cases

[Brian E Carpenter <[e-mail]>]

On 07/09/2015 14:26, Sheng Jiang wrote:
> 
> 
>>>> DDoS and APT are very active research topics. Application layer DDoS
>>>> attacks are more difficult to detect than layer 4 DDoS attacks. In many
>>>> cases, the application layer DDoS does not introduce large amount
>>>> traffics. However, by using big data and data mining tech, it is possible
>>>> to find out the clues of such attacks.
>>>
>>> Hi, Dacheng,
>>>
>>> Applying machine learning in DDoS protection is an interest use case. For my
>> understanding, the machine would learn the potential attack behaviors, am I
>> right?
>>>
>>> If yes, I have two questions: a) does the machine learning has the possibility
>> to learn/identify new attack behaviors, which was not recognized before? If
>> yes, what is the working principles? b) is it possible for autonomic reaction
>>from the network operational perspective after detect such DDoS attack?
>> Give the machine learning may not be accurate, my guess is human
>> intervention is needed.
>>
>> I suppose this is in some ways similar to spam processing in email. For
>> example,
>> a Bayesian spam filter is surprisingly good, but sometimes makes mistakes, so
>> human training to correct false positives and false negatives is essential.
> 
> Hi, Brian,
> 
> I believe there is feedback-style training in spam processing. But what do you mean by "human" training? Do you mean the feedback is decided and feed by human administrators? I believe this could be done by machine learning mechanisms.

In the spam case it is definitely a real live human who must detect an error
by the classifier. In my case I estimate that maybe 0.5% of the messages
in my Gmail spam folder are not spam, and maybe 1% in my Gmail Inbox are actually
spam.

I think that for real-time DDOS protection, the solution has to include real-time
input from an operator for both cases: "This is not an attack" and "You missed
an attack". I think that means that the machine-learning system will always
run in training mode, even if training is only needed in 1% of cases.

> 
> Actually, spam filtering was one of the earliest network-relevant area that starts to use machine learning. It would worth to study the machine learning applications in spam filtering. Or we could invite some expert in this area to join nmlrg discussion.

Agreed
  Brian

> 
> Best regards,
> 
> Sheng
> 
>> How can
>> human training be achieved for a real-time case like DDOS?
>>
>>   Brian
>>
>>>
>>> Best regards,
>>>
>>> Sheng
>>>
>>>> There were some related discussions in Dots. If you are interested, I
>>>> could find them out later.
>>>>
>>>> _______________________________________________
>>>> nmlrg mailing list
>>>> [e-mail]
>>>> https://www.irtf.org/mailman/listinfo/nmlrg
>>> _______________________________________________
>>> nmlrg mailing list
>>> [e-mail]
>>> https://www.irtf.org/mailman/listinfo/nmlrg
>>>