Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

David Meyer <dmm@1-4-5.net> Wed, 29 March 2017 14:58 UTC

Return-Path: <dmm@1-4-5.net>
X-Original-To: idnet@ietfa.amsl.com
Delivered-To: idnet@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4ED401287A5 for <idnet@ietfa.amsl.com>; Wed, 29 Mar 2017 07:58:12 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.898
X-Spam-Level:
X-Spam-Status: No, score=-1.898 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=1-4-5-net.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6BWbqXYGs0cK for <idnet@ietfa.amsl.com>; Wed, 29 Mar 2017 07:58:09 -0700 (PDT)
Received: from mail-qt0-x234.google.com (mail-qt0-x234.google.com [IPv6:2607:f8b0:400d:c0d::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8D130129552 for <idnet@ietf.org>; Wed, 29 Mar 2017 07:58:02 -0700 (PDT)
Received: by mail-qt0-x234.google.com with SMTP id x35so14758905qtc.2 for <idnet@ietf.org>; Wed, 29 Mar 2017 07:58:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1-4-5-net.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=P6XCFz6Lp60p6am/Xii3mdVZ/Hluh6JESHu1zBnoaak=; b=bs15Uif9AuyD5RLUGBBauIbzkPyTDCdiRCNCMGXXd6X4iNkwtp7ybDwNB36jzcLouA fTg3nX45ZM1YsdENDtOfXaJ57Oz9yVZR1LgqFBzZFUZ0U0pAycvPWGTIpYe5Kl92Yz2a uGrCCeiMNged0OdVLhSHiuZtflqySQOxst1NDZC1WffV73ozZIm0L6ixjBZX/9EAwe+u egv3cr1n0K/JdMEdRvNSADte/TklmDLYKgtF3mBdSQtfSVD+D//9JU79mXdFY9f1WEeC faEepc11isaNJaAHaEiUmBVtpfLSR4WoKNQfuVdHE+cTWbHzdGpLTkidY+aghrt4dIIX shqQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=P6XCFz6Lp60p6am/Xii3mdVZ/Hluh6JESHu1zBnoaak=; b=jsRFV58S3aSKLR7GPIw/zrDeGMUw5+5TqOQXaJU1+JZJEKZD5OkaGRVWImvcJD3z7h ZJDJCuN8eR6+Tqr3IZ5Yf5GXAZf1KKDSeuTWJ2A2m/jbwaKz4nAZX/d6t7Z8r9RAsNBs zcLrAcJL6z5k+aQXDREcQeSoirv+Bnr5jJNw4n88claXIThYumxMcnnKKU/SOSyi5Zic 6e/DR1oBoW01//TrdUyuu4dqsMkqzrdZdMBTC8Ybx4xTvgqvNtZz+KoTgQ4keQZDkzXz x/i5x+02piVCwMlyKe8uujoVOBhUtETMpOEOovFmL/9mzES61rtfviQh+gWZDMCZ3KCP kFDw==
X-Gm-Message-State: AFeK/H1+9zaILWGC7Ehrq8W1vqXVAj1U6DSQfeuQ0SBu7v7EsWgF2TYF4gXbkTcwJ6Vm204u4DKpWmWVKNGCow==
X-Received: by 10.237.38.229 with SMTP id q92mr1033296qtd.6.1490799481182; Wed, 29 Mar 2017 07:58:01 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.12.149.34 with HTTP; Wed, 29 Mar 2017 07:58:00 -0700 (PDT)
X-Originating-IP: [128.223.156.253]
In-Reply-To: <CAAVmtwez1E+VS2XOBpf8bXsj98VrNn8DS7sTUcPJ_-VOfQfshQ@mail.gmail.com>
References: <5D36713D8A4E7348A7E10DF7437A4B927CD15A18@NKGEML515-MBS.china.huawei.com> <CAHiKxWgT3hKr2VwhbfpmR_siHgiY4PbiKy3QgesG7uqUTnedmw@mail.gmail.com> <20170328182530.GP4808@spectre> <84ff06ac-61e7-3e71-f6f9-c78335c69aaf@inria.fr> <CAAVmtwez1E+VS2XOBpf8bXsj98VrNn8DS7sTUcPJ_-VOfQfshQ@mail.gmail.com>
From: David Meyer <dmm@1-4-5.net>
Date: Wed, 29 Mar 2017 07:58:00 -0700
Message-ID: <CAHiKxWgyTAotoKPmCHFsfpZfmWgOsANTgWqnLL=RMJvifocmPQ@mail.gmail.com>
To: Chris Hammerschmidt <laxris@gmail.com>
Cc: =?UTF-8?B?SsOpcsO0bWUgRnJhbsOnb2lz?= <jerome.francois@inria.fr>, list-idnet <idnet@ietf.org>
Content-Type: multipart/alternative; boundary=94eb2c0bb94e826250054bdfcded
Archived-At: <https://mailarchive.ietf.org/arch/msg/idnet/xm88TYwzVWlMvWwDJ2UjwSQx_EI>
Subject: Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests
X-BeenThere: idnet@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "The IDNet \(Intelligence-Defined Network\) " <idnet.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/idnet>, <mailto:idnet-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/idnet/>
List-Post: <mailto:idnet@ietf.org>
List-Help: <mailto:idnet-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/idnet>, <mailto:idnet-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 29 Mar 2017 14:58:12 -0000

Chris,

Totally agree. This is part of the reason we need standardized data sets,
namely, so we can compare results and understand if we're making progress.
See e.g., http://image-net.org/challenges/LSVRC/2016/ for an example.  The
result has been the steady ratcheting down of error rates to super-human
levels. We have nothing like this for networking, and as I have been
saying, that is one of our biggest challenges.

Dave


On Wed, Mar 29, 2017 at 7:33 AM, Chris Hammerschmidt <laxris@gmail.com>
wrote:

> Hi,
>
> equally important to having datasets* is proposing benchmarks with the
> datasets.* Right now, everyone has different expectations and from my
> personal experience, many reviewers at networking conferences won't accept
> a "typical" machine learning paper with applications in networking
> consisting of data+model+evaluation. Expectations range from something like
> a large-scale practical application case to addressing worries about
> adversarial influence on the learned model, or systems that take not only
> care of machine-learning malicious/anomalous/interesting behavior but
> want to have guidelines how to apply it, e.g. filtering rules and
> parameter/threshold setting guidelines. Often, these values have to be set
> with domain knowledge, adjusted to the specific application case (as
> networks can be very different).
>
> The lack of benchmarks makes it incredibly hard to compare different
> solutions to each other, or even to just apply a new method to an old
> problem; and indeed, reviewers have pointed out to me that the problem of
> machine learning from network traffic has been solved already and there is
> no point in further papers on this topic. A *shared benchmark* solves
> this problem by delegating the argument for the relevance ONCE while
> setting up the benchmark, rather than distributing the task to each author.
>
> Cheers
> Christian
>
> On 28 March 2017 at 20:46, Jérôme François <jerome.francois@inria.fr>
> wrote:
>
>> Hi
>>
>> Le 28/03/2017 à 20:25, Pedro Martinez-Julia a écrit :
>> > On Tue, Mar 28, 2017 at 10:59:38AM -0700, David Meyer wrote:
>> >> Hey Sheng,
>> >>
>> >> I just wanted to revive my key concern on [0] (same one I made at the
>> >> NMRL): The hard parts of getting Machine Learning intelligence into
>> >> Networking is the Machine Learning part. In addition, successful
>> deployment
>> >> of ML requires knowledge of ML combined with domain knowledge. We
>> >> definitely have the domain knowledge; the problem is that we don't
>> have the
>> >> ML knowledge, and this is one of the big factors holding us back; see
>> e.g.
>> >> Andrew's discussion of talent in [1].  Slides such as [0] seem to imply
>> >> that *someone else* (in particular, not us)  will handle the ML part
>> of all
>> >> of this. I'll just note that in general successful deployments of ML
>> don't
>> >> work this way; the domain experts will have to learn ML (and vice
>> versa)
>> >> for us to be successful (again, see [1] and many others).
>> > Dear Dave,
>> >
>> > You are true in that ML/domain knowledge is necessary but, however it is
>> > worth to take into account that it is not strictly required and it will
>> > even be counterproductive in some (or maybe most) situations. At the end
>> > of the day, encouraging (or forcing) a network expert to learn ML is
>> > quite difficult, the results will be delayed until the learning phase
>> > ends, and (most probably) s/he will never get a better solution than a
>> > person that has been an expert in ML from a long time ago. Therefore, it
>> > is better to make separate experts (in ML and the domain itself) to
>> > collaborate in a common solution. Therefore, and I think it has been
>> > mentioned before, we have to (try to) enroll experts in ML to the IDNET
>> > group and see what can we do together...
>> This is a general trend that only a single person cannot be expert in
>> everything. Actually, a good network expert may require good ML but also
>> good software skills (including software formal verification knowledge).
>> So, in my opinion the problem is larger.
>>
>> Enhancing collaboration between network and ML expert is a path that
>> starts in many company and insitutes I think. Discussing with ML
>> experts, they are usually open and happy to discover new "use cases"
>> but their first question will be "do you have some labelled datasets
>> that we can work with" which relates to the problem raised in previous
>> emails about open datasets.
>>
>> In my opinion, if we wan to attract ML experts in our dicussions, we
>> should identify few scenrios, defined them precisely and provide an open
>> dataset. By defining them, I mean we have to give them all background
>> they need to understand (that can be built incrementally through
>> discussion) in a well-documented format.
>>
>> jerome
>>
>> >> Perhaps a useful exercise would be to write an ID that makes your
>> >> assumptions explicit?
>> >>
>> >> Thanks,
>> >> Dave
>> > Regards,
>> > Pedro
>> >
>>
>>
>> _______________________________________________
>> IDNET mailing list
>> IDNET@ietf.org
>> https://www.ietf.org/mailman/listinfo/idnet
>>
>
>
> _______________________________________________
> IDNET mailing list
> IDNET@ietf.org
> https://www.ietf.org/mailman/listinfo/idnet
>
>