Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

David Meyer <> Wed, 29 March 2017 14:58 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 4ED401287A5 for <>; Wed, 29 Mar 2017 07:58:12 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.898
X-Spam-Status: No, score=-1.898 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 6BWbqXYGs0cK for <>; Wed, 29 Mar 2017 07:58:09 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:400d:c0d::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 8D130129552 for <>; Wed, 29 Mar 2017 07:58:02 -0700 (PDT)
Received: by with SMTP id x35so14758905qtc.2 for <>; Wed, 29 Mar 2017 07:58:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=P6XCFz6Lp60p6am/Xii3mdVZ/Hluh6JESHu1zBnoaak=; b=bs15Uif9AuyD5RLUGBBauIbzkPyTDCdiRCNCMGXXd6X4iNkwtp7ybDwNB36jzcLouA fTg3nX45ZM1YsdENDtOfXaJ57Oz9yVZR1LgqFBzZFUZ0U0pAycvPWGTIpYe5Kl92Yz2a uGrCCeiMNged0OdVLhSHiuZtflqySQOxst1NDZC1WffV73ozZIm0L6ixjBZX/9EAwe+u egv3cr1n0K/JdMEdRvNSADte/TklmDLYKgtF3mBdSQtfSVD+D//9JU79mXdFY9f1WEeC faEepc11isaNJaAHaEiUmBVtpfLSR4WoKNQfuVdHE+cTWbHzdGpLTkidY+aghrt4dIIX shqQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=P6XCFz6Lp60p6am/Xii3mdVZ/Hluh6JESHu1zBnoaak=; b=jsRFV58S3aSKLR7GPIw/zrDeGMUw5+5TqOQXaJU1+JZJEKZD5OkaGRVWImvcJD3z7h ZJDJCuN8eR6+Tqr3IZ5Yf5GXAZf1KKDSeuTWJ2A2m/jbwaKz4nAZX/d6t7Z8r9RAsNBs zcLrAcJL6z5k+aQXDREcQeSoirv+Bnr5jJNw4n88claXIThYumxMcnnKKU/SOSyi5Zic 6e/DR1oBoW01//TrdUyuu4dqsMkqzrdZdMBTC8Ybx4xTvgqvNtZz+KoTgQ4keQZDkzXz x/i5x+02piVCwMlyKe8uujoVOBhUtETMpOEOovFmL/9mzES61rtfviQh+gWZDMCZ3KCP kFDw==
X-Gm-Message-State: AFeK/H1+9zaILWGC7Ehrq8W1vqXVAj1U6DSQfeuQ0SBu7v7EsWgF2TYF4gXbkTcwJ6Vm204u4DKpWmWVKNGCow==
X-Received: by with SMTP id q92mr1033296qtd.6.1490799481182; Wed, 29 Mar 2017 07:58:01 -0700 (PDT)
MIME-Version: 1.0
Received: by with HTTP; Wed, 29 Mar 2017 07:58:00 -0700 (PDT)
X-Originating-IP: []
In-Reply-To: <>
References: <> <> <20170328182530.GP4808@spectre> <> <>
From: David Meyer <>
Date: Wed, 29 Mar 2017 07:58:00 -0700
Message-ID: <>
To: Chris Hammerschmidt <>
Cc: =?UTF-8?B?SsOpcsO0bWUgRnJhbsOnb2lz?= <>, list-idnet <>
Content-Type: multipart/alternative; boundary=94eb2c0bb94e826250054bdfcded
Archived-At: <>
Subject: Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: "The IDNet \(Intelligence-Defined Network\) " <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 29 Mar 2017 14:58:12 -0000


Totally agree. This is part of the reason we need standardized data sets,
namely, so we can compare results and understand if we're making progress.
See e.g., for an example.  The
result has been the steady ratcheting down of error rates to super-human
levels. We have nothing like this for networking, and as I have been
saying, that is one of our biggest challenges.


On Wed, Mar 29, 2017 at 7:33 AM, Chris Hammerschmidt <>

> Hi,
> equally important to having datasets* is proposing benchmarks with the
> datasets.* Right now, everyone has different expectations and from my
> personal experience, many reviewers at networking conferences won't accept
> a "typical" machine learning paper with applications in networking
> consisting of data+model+evaluation. Expectations range from something like
> a large-scale practical application case to addressing worries about
> adversarial influence on the learned model, or systems that take not only
> care of machine-learning malicious/anomalous/interesting behavior but
> want to have guidelines how to apply it, e.g. filtering rules and
> parameter/threshold setting guidelines. Often, these values have to be set
> with domain knowledge, adjusted to the specific application case (as
> networks can be very different).
> The lack of benchmarks makes it incredibly hard to compare different
> solutions to each other, or even to just apply a new method to an old
> problem; and indeed, reviewers have pointed out to me that the problem of
> machine learning from network traffic has been solved already and there is
> no point in further papers on this topic. A *shared benchmark* solves
> this problem by delegating the argument for the relevance ONCE while
> setting up the benchmark, rather than distributing the task to each author.
> Cheers
> Christian
> On 28 March 2017 at 20:46, Jérôme François <>
> wrote:
>> Hi
>> Le 28/03/2017 à 20:25, Pedro Martinez-Julia a écrit :
>> > On Tue, Mar 28, 2017 at 10:59:38AM -0700, David Meyer wrote:
>> >> Hey Sheng,
>> >>
>> >> I just wanted to revive my key concern on [0] (same one I made at the
>> >> NMRL): The hard parts of getting Machine Learning intelligence into
>> >> Networking is the Machine Learning part. In addition, successful
>> deployment
>> >> of ML requires knowledge of ML combined with domain knowledge. We
>> >> definitely have the domain knowledge; the problem is that we don't
>> have the
>> >> ML knowledge, and this is one of the big factors holding us back; see
>> e.g.
>> >> Andrew's discussion of talent in [1].  Slides such as [0] seem to imply
>> >> that *someone else* (in particular, not us)  will handle the ML part
>> of all
>> >> of this. I'll just note that in general successful deployments of ML
>> don't
>> >> work this way; the domain experts will have to learn ML (and vice
>> versa)
>> >> for us to be successful (again, see [1] and many others).
>> > Dear Dave,
>> >
>> > You are true in that ML/domain knowledge is necessary but, however it is
>> > worth to take into account that it is not strictly required and it will
>> > even be counterproductive in some (or maybe most) situations. At the end
>> > of the day, encouraging (or forcing) a network expert to learn ML is
>> > quite difficult, the results will be delayed until the learning phase
>> > ends, and (most probably) s/he will never get a better solution than a
>> > person that has been an expert in ML from a long time ago. Therefore, it
>> > is better to make separate experts (in ML and the domain itself) to
>> > collaborate in a common solution. Therefore, and I think it has been
>> > mentioned before, we have to (try to) enroll experts in ML to the IDNET
>> > group and see what can we do together...
>> This is a general trend that only a single person cannot be expert in
>> everything. Actually, a good network expert may require good ML but also
>> good software skills (including software formal verification knowledge).
>> So, in my opinion the problem is larger.
>> Enhancing collaboration between network and ML expert is a path that
>> starts in many company and insitutes I think. Discussing with ML
>> experts, they are usually open and happy to discover new "use cases"
>> but their first question will be "do you have some labelled datasets
>> that we can work with" which relates to the problem raised in previous
>> emails about open datasets.
>> In my opinion, if we wan to attract ML experts in our dicussions, we
>> should identify few scenrios, defined them precisely and provide an open
>> dataset. By defining them, I mean we have to give them all background
>> they need to understand (that can be built incrementally through
>> discussion) in a well-documented format.
>> jerome
>> >> Perhaps a useful exercise would be to write an ID that makes your
>> >> assumptions explicit?
>> >>
>> >> Thanks,
>> >> Dave
>> > Regards,
>> > Pedro
>> >
>> _______________________________________________
>> IDNET mailing list
> _______________________________________________
> IDNET mailing list