Re: [Nmlrg] links to each slide of presentations//RE: slides of NMLRG #3, June 27th, Athens

[Sheng Jiang <[e-mail]>]

Hi, David,



Thanks so much for your email. I am sharing the same concern you have. Actually, this Monday in our NMLRG #3, Athens, we did discussed this issue regarding to the potential standardized dataset. And we are planning to have an open discussion to regarding to the requirements and validation qualification of the potential common dataset in the coming NMLRG #4, Berlin, IETF96.



However, upto now, I have serveral concerns. Firstly, before we could discuss such dataset, we need to narrow our target scenarios to a few very specific use cases, giving that there are so many various scenarios in the network area and they are so different from each other and require different dataset. Secondly, as we did talk with operators, the real data is the important properties for them. So, it would be almost impossible for them to share them. It leaves two possibilities on how to get dataset: a) generating dataset using simulation; b) try to get some partial dataset from operators that have removed their sensenble data. Both way are problemstic, in the risk that a lot of efforts may waste in wrong areas. Thirdly, the availability of data may very different in different network environments. It depends on the measurement/congnitive functions that have been deployed or implemented.



Although having the above concerns, I  think we should: a) research on certain dataset. It would be very useful to prove the machine learning could perform better or provide more flexibility and adaptibility; b) continue our discuss on various network use cases that could benefit from applying machine learning, in order to serve wider network communities.



Another important point I would like to discuss is that you mentioned "our goal is to provide accurate, repeatable, and explainable results." Ideally, we do want this. It is our traditional way for logic-based programming. However, I am not sure that is a doable target with machine learning. Many of machine learning algorithms are based on statistics theory, which may not be that much accurate. Some of machine learning algorithms are black box or have very complexicated internal computing process. So, the resulf of these algorithms may be hard to be explained. Also the process of machine learning may be hard to be controlled or intervened. Repeatable, yes, we have to have this feature to be widely adopted. However, repeatable may have to come with some level of toleration for the inaccurate.



Best regards,



Sheng



________________________________

From: David Meyer [[e-mail]]
Sent: 28 June 2016 23:30
To: Sheng Jiang
Cc: [e-mail]
Subject: Re: [Nmlrg] links to each slide of presentations//RE: slides of NMLRG #3, June 27th, Athens

Sheng,

Thanks for the pointers. One thing I notice is that we don't have much on what the characteristics of network data might be, and as such, what kinds of existing learning algorithms might be suitable and where additional development might be required. For example, is the collected data IID? Is it time series? Is the underlying distribution stationary? If not, how does that constrain algorithms we might use or develop? For example, how does a given algorithm deal with concept drift/internal covariate shift (in the case of DNNs; see e.g., batch normalization [0]). There are many other such questions, such as is the data categorical (e.g., ports, IP addresses) or is it continuous/discrete (e.g., counters). And if the data in question is categorical, what is the cardinality of the categories (this will inform how such data can be encoded); in the case of IP addresses we can't really one-hot encode addresses because their cardinality is too large (2^32 or 2^128); this has implications for how we build classifiers (in particular, for softmax layers in DNNs of various kinds).

Related to the above is the question of features? What are good features for networking? Where do they come from? Are they domain specific? Can we learn features the way a DNNs does in the network space? Can we use autoencoders to discover such features? Or can we use GANs to train DNNs for network classification tasks in an unsupervised manner? Are there other, non-ad-hoc (well founded) methods we can use, or is every use case a one-off (one would hope not).

We can carry the same kinds of analyzes to the algorithms applied. For example, while something k-means is an effective way to get a feeling for how continuous/discrete hangs together, if our data is categorical statistical clustering approaches such as LDA might provide a more well-founded approach (of course, as with most Bayesian techniques is the question of approximate inference arises since in most interesting cases the integral that we need to solve, namely the marginal probability data isn't tractable so we need to resort to MCMC or more likely variational inference). And what about the use of SGD/batch normalization etc with DNNs, and perhaps more importantly, can we use network data to train DNN policy networks for reinforcement learning like we saw in deep Q-learning and AlphaGo?

These comments are all by way of saying that we don't have a solid theoretical understanding (yet) of how techniques that have been so successful in other domains (e.g., DNNs for perceptual tasks) generalize to networking use cases. We will need this understanding if our goal is to provide accurate, repeatable, and explainable results.

In order to accomplish all of this we need, as I have been saying , not only a good understanding of how these algorithms work but also standardized data sets and associated benchmarks so we can tell if we are making progress (or even if our techniques work). Analogies here include MNIST and ImageNet and their associated benchmarks, among others. As mentioned  standardized data sets are key to making progress in the ML for networking space (otherwise how do you know your technique works and/or improves on another techniques?). One might assume that these data sets would need to be labeled (as supervised learning is where most of the progress is being made these days), but not necessarily; Generative Adversarial Networks (GANs) have emerged as a new way to train DNNs in an unsupervised manner (this is moving very rapidly; see e.g., https://openai.com/blog/generative-models/).

The summary here is that the "distance" between theory and practice in ML is effectively zero right now due to the incredible rate of progress in the field; this means we need  to understand both sides of the theory/practice coin in order to be effective. None of the slide decks provide much background on what the proposed algorithms are, how they work, or why they should be expected to work on network data.

Finally, if you are interested in LDA or other algorithms there are a few short explanatory pieces I have written for my team on http://www.1-4-5.net/~dmm/ml (works in progress).

Thanks,

Dave

[0] https://arxiv.org/pdf/1502.03167.pdf

On Tue, Jun 28, 2016 at 7:03 AM, Sheng Jiang <[e-mail]<mailto:[e-mail]>> wrote:
Oops... The proceeding page for interim meeting seems not be as intelligent as the proceeding pages of IETF meetings. Our proceeding page does not autonomically show the slides. I have sent an email to IETF secretary to ask them to fix it. Meanwhile, in this email, here is the links for each presentations:

Slide Filename Edit Replace Delete
Chair Slides
https://www.ietf.org/proceedings/interim-2016-nmlrg-01/slides/slides-interim-2016-nmlrg-01-0.pdf

Introduction to Network Machine Learning & NMLRG
https://www.ietf.org/proceedings/interim-2016-nmlrg-01/slides/slides-interim-2016-nmlrg-01-1.pdf

Data Collection and Analysis At High Security Lab
https://www.ietf.org/proceedings/interim-2016-nmlrg-01/slides/slides-interim-2016-nmlrg-01-2.pdf

Use Cases of Applying Machine Learning Mechanism with Network Traffic
https://www.ietf.org/proceedings/interim-2016-nmlrg-01/slides/slides-interim-2016-nmlrg-01-3.pdf

Mobile network state characterization and prediction
https://www.ietf.org/proceedings/interim-2016-nmlrg-01/slides/slides-interim-2016-nmlrg-01-4.pdf

Learning how to route
https://www.ietf.org/proceedings/interim-2016-nmlrg-01/slides/slides-interim-2016-nmlrg-01-5.pdf

Regards,

Sheng
________________________________________
From: nmlrg [[e-mail]<mailto:[e-mail]>] on behalf of Sheng Jiang [[e-mail]<mailto:[e-mail]>]
Sent: 28 June 2016 21:03
To: [e-mail]<mailto:[e-mail]>
Subject: [Nmlrg] slides of NMLRG #3, June 27th, Athens

Hi, nmlrg,

All slides that have been presented in our NMLRG #3 meeting, June 27th, 2016, Athens, Greece, with EUCNC2016, have been uploaded. They can be accessed through below link

https://www.ietf.org/proceedings/interim/2016/06/27/nmlrg/proceedings.html

Best regards,

Sheng
_______________________________________________
nmlrg mailing list
[e-mail]<mailto:[e-mail]>
https://www.irtf.org/mailman/listinfo/nmlrg
_______________________________________________
nmlrg mailing list
[e-mail]<mailto:[e-mail]>
https://www.irtf.org/mailman/listinfo/nmlrg