Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

Chris Hammerschmidt <laxris@gmail.com> Wed, 29 March 2017 14:34 UTC

Return-Path: <laxris@gmail.com>
X-Original-To: idnet@ietfa.amsl.com
Delivered-To: idnet@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 34374129521 for <idnet@ietfa.amsl.com>; Wed, 29 Mar 2017 07:34:00 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.698
X-Spam-Level:
X-Spam-Status: No, score=-2.698 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wAOWSjiXRghM for <idnet@ietfa.amsl.com>; Wed, 29 Mar 2017 07:33:52 -0700 (PDT)
Received: from mail-qk0-x22d.google.com (mail-qk0-x22d.google.com [IPv6:2607:f8b0:400d:c09::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2D9B2129541 for <idnet@ietf.org>; Wed, 29 Mar 2017 07:33:38 -0700 (PDT)
Received: by mail-qk0-x22d.google.com with SMTP id f11so14367557qkb.0 for <idnet@ietf.org>; Wed, 29 Mar 2017 07:33:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=MdRXJhzmAazzrqrKJJpvHODES9D390Gf7p5YwD8Bbgg=; b=kGPXUWtvnW+F6WF3ZoDMwNlvSoLNFOBQSmmy4h9Q+aNFa3Sr3LZCOG3+e2qeghFU7a vAWTt5ATATbBqJ1MDKpWk9LRa3Bn7E/gP/g4vuAJWDU9ZQMOswM75x03qbl26cGcXKJI L7rrllyF/9TB5QpfIOpoipmn9buQwBBkfFanERmnZb9I96n8vMicTHPmuQGP/aCXLKaU GQkM8ygpmqbWPYYaz7mfYYIlsQl274YvrYsgkLoD1VERlaaq0yjpUbW+jzm/SE+G+37I aE4SD4V0eUDBMnO2+iOQwyIyg1kpIka57kdqS7n1+spKXiVhXdHPwC4h2XgCzWYJere0 blZA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=MdRXJhzmAazzrqrKJJpvHODES9D390Gf7p5YwD8Bbgg=; b=ZkcRjR4Qp/jqvKWUCqJqFP9ZsSuB4DgXUy86g9LSUy6LpV9+/OmqoNgWW4tx9GbWuY KLKh/A7T94heCtZ+bXWY9qjiOS9ToVewqKVETr4K0jCeT+wBDCHkrfkJ8GdENg93vorI 2lltuvYSYqHfT/WL89WhddgKN/Ini1o+CcKe/bSnQF4d5P0jKfPZgL9QlDcuTWUkW85G R1AUUkFjrfSmPoQJXzdFND79NS/HMRPb9nLxsXHvZh3c8FRpPe5qETCeRGilhLvOoD9q AhpcB/4xDcpHGc8Y88JP/TaHwPd+58DEv964z2XFUtQJLzPJuJrr0iqsGfdqcSBFTfg1 Zhyg==
X-Gm-Message-State: AFeK/H0dKiUjWVMPKq3dWsrBRBa5oFiJ1cMbYhQhvlXN+rtsqwM569q1DmrTvUqDbV9kY+DtWzUMn25iqZsX7A==
X-Received: by 10.55.75.146 with SMTP id y140mr342304qka.200.1490798017068; Wed, 29 Mar 2017 07:33:37 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.140.97.163 with HTTP; Wed, 29 Mar 2017 07:33:36 -0700 (PDT)
In-Reply-To: <84ff06ac-61e7-3e71-f6f9-c78335c69aaf@inria.fr>
References: <5D36713D8A4E7348A7E10DF7437A4B927CD15A18@NKGEML515-MBS.china.huawei.com> <CAHiKxWgT3hKr2VwhbfpmR_siHgiY4PbiKy3QgesG7uqUTnedmw@mail.gmail.com> <20170328182530.GP4808@spectre> <84ff06ac-61e7-3e71-f6f9-c78335c69aaf@inria.fr>
From: Chris Hammerschmidt <laxris@gmail.com>
Date: Wed, 29 Mar 2017 16:33:36 +0200
Message-ID: <CAAVmtwez1E+VS2XOBpf8bXsj98VrNn8DS7sTUcPJ_-VOfQfshQ@mail.gmail.com>
To: =?UTF-8?B?SsOpcsO0bWUgRnJhbsOnb2lz?= <jerome.francois@inria.fr>
Cc: idnet@ietf.org
Content-Type: multipart/alternative; boundary=001a114819443d62a3054bdf76f4
Archived-At: <https://mailarchive.ietf.org/arch/msg/idnet/TT5BiKEYwxwUZPER26aHfSbmVQQ>
Subject: Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests
X-BeenThere: idnet@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "The IDNet \(Intelligence-Defined Network\) " <idnet.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/idnet>, <mailto:idnet-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/idnet/>
List-Post: <mailto:idnet@ietf.org>
List-Help: <mailto:idnet-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/idnet>, <mailto:idnet-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 29 Mar 2017 14:34:00 -0000

Hi,

equally important to having datasets* is proposing benchmarks with the
datasets.* Right now, everyone has different expectations and from my
personal experience, many reviewers at networking conferences won't accept
a "typical" machine learning paper with applications in networking
consisting of data+model+evaluation. Expectations range from something like
a large-scale practical application case to addressing worries about
adversarial influence on the learned model, or systems that take not only
care of machine-learning malicious/anomalous/interesting behavior but want
to have guidelines how to apply it, e.g. filtering rules and
parameter/threshold setting guidelines. Often, these values have to be set
with domain knowledge, adjusted to the specific application case (as
networks can be very different).

The lack of benchmarks makes it incredibly hard to compare different
solutions to each other, or even to just apply a new method to an old
problem; and indeed, reviewers have pointed out to me that the problem of
machine learning from network traffic has been solved already and there is
no point in further papers on this topic. A *shared benchmark* solves this
problem by delegating the argument for the relevance ONCE while setting up
the benchmark, rather than distributing the task to each author.

Cheers
Christian

On 28 March 2017 at 20:46, Jérôme François <jerome.francois@inria.fr> wrote:

> Hi
>
> Le 28/03/2017 à 20:25, Pedro Martinez-Julia a écrit :
> > On Tue, Mar 28, 2017 at 10:59:38AM -0700, David Meyer wrote:
> >> Hey Sheng,
> >>
> >> I just wanted to revive my key concern on [0] (same one I made at the
> >> NMRL): The hard parts of getting Machine Learning intelligence into
> >> Networking is the Machine Learning part. In addition, successful
> deployment
> >> of ML requires knowledge of ML combined with domain knowledge. We
> >> definitely have the domain knowledge; the problem is that we don't have
> the
> >> ML knowledge, and this is one of the big factors holding us back; see
> e.g.
> >> Andrew's discussion of talent in [1].  Slides such as [0] seem to imply
> >> that *someone else* (in particular, not us)  will handle the ML part of
> all
> >> of this. I'll just note that in general successful deployments of ML
> don't
> >> work this way; the domain experts will have to learn ML (and vice versa)
> >> for us to be successful (again, see [1] and many others).
> > Dear Dave,
> >
> > You are true in that ML/domain knowledge is necessary but, however it is
> > worth to take into account that it is not strictly required and it will
> > even be counterproductive in some (or maybe most) situations. At the end
> > of the day, encouraging (or forcing) a network expert to learn ML is
> > quite difficult, the results will be delayed until the learning phase
> > ends, and (most probably) s/he will never get a better solution than a
> > person that has been an expert in ML from a long time ago. Therefore, it
> > is better to make separate experts (in ML and the domain itself) to
> > collaborate in a common solution. Therefore, and I think it has been
> > mentioned before, we have to (try to) enroll experts in ML to the IDNET
> > group and see what can we do together...
> This is a general trend that only a single person cannot be expert in
> everything. Actually, a good network expert may require good ML but also
> good software skills (including software formal verification knowledge).
> So, in my opinion the problem is larger.
>
> Enhancing collaboration between network and ML expert is a path that
> starts in many company and insitutes I think. Discussing with ML
> experts, they are usually open and happy to discover new "use cases"
> but their first question will be "do you have some labelled datasets
> that we can work with" which relates to the problem raised in previous
> emails about open datasets.
>
> In my opinion, if we wan to attract ML experts in our dicussions, we
> should identify few scenrios, defined them precisely and provide an open
> dataset. By defining them, I mean we have to give them all background
> they need to understand (that can be built incrementally through
> discussion) in a well-documented format.
>
> jerome
>
> >> Perhaps a useful exercise would be to write an ID that makes your
> >> assumptions explicit?
> >>
> >> Thanks,
> >> Dave
> > Regards,
> > Pedro
> >
>
>
> _______________________________________________
> IDNET mailing list
> IDNET@ietf.org
> https://www.ietf.org/mailman/listinfo/idnet
>