Re: [ippm] How should capacity measurement interact with shaping?

Matt Mathis <mattmathis@google.com> Mon, 23 September 2019 18:36 UTC

Return-Path: <mattmathis@google.com>
X-Original-To: ippm@ietfa.amsl.com
Delivered-To: ippm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 69B0D120088 for <ippm@ietfa.amsl.com>; Mon, 23 Sep 2019 11:36:22 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.5
X-Spam-Level:
X-Spam-Status: No, score=-17.5 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lu8hZT5WWvFl for <ippm@ietfa.amsl.com>; Mon, 23 Sep 2019 11:36:19 -0700 (PDT)
Received: from mail-yb1-xb31.google.com (mail-yb1-xb31.google.com [IPv6:2607:f8b0:4864:20::b31]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id BA5A1120020 for <ippm@ietf.org>; Mon, 23 Sep 2019 11:36:18 -0700 (PDT)
Received: by mail-yb1-xb31.google.com with SMTP id f1so5676637ybq.11 for <ippm@ietf.org>; Mon, 23 Sep 2019 11:36:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=WvJzVbksanox10K0g8pYUJlibX+DNhHjQDCgqees5M4=; b=vOlE6pLyFiSKQfkYzHJD8+tV3glerK/ZuwBbLfKwtCzwCAalvaapFJ44jcrD1hAjwy NqPPgmJdLSnQTmhpjqCJpYXW7VQX/0yWcL6DLu0lKPirZJtzT3cMoSELjWyvgdhvCx46 zyqcqIxzrNaq/XNYWlTsqxcubo/2DqYvLLjlJKUOdIO3mAxOVBdB208EEOmcShLNjqWD d+44oLwtgxGLhGNmFPFmKN0KlTUX0tztutEKWbl6wG7K0c2/0FExsVgXxV1Br8ijqhXl Rp5chtiyXTkeKHs5UTNWXgGTvjKjr3F/2F3R66i4gE5sPcsHpBQNYPDIYmqPtApecr36 yLFA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=WvJzVbksanox10K0g8pYUJlibX+DNhHjQDCgqees5M4=; b=LwoGWjk1hkRF1ATSoNnSuLHgH7tIb2jdy3lhdTD/L7g4vVIC3caSykzCmTjRIqB2mZ EzsTn+BmCt4HRrr43OVe3A7kk340zO6TT/Wuvw6qmjTS9Wo3Tc7lXAaV7bhTRUPGHaO2 aHnIf7lYV6xGjR/iN9q/YaQ+32r1SbkNkVm+yjO4wi3ZE4V8kwBhqHawIz4zVclgmUYf uKqpJ3pz6z5IkHFNQKw+9kn5Tlr7XglEfi149tGuI5Mu2AJ6jy22zPyOVq6ZMAXhBqem i1dKSYC2+0H4Gtm+Kpiqn19waMe3ZlI6/MWj1jN7mI5zux+oDzW6mG2K/cQO6++jIJyW MBKA==
X-Gm-Message-State: APjAAAWJ5eKtvhUpMQAspYNM/XtYrZPa5xaK/3rshx5SMYzi8goO6u9J FiGe+Ajx6614YgM8yHXClmSyS5LeswuXSa7dFZVujQ==
X-Google-Smtp-Source: APXvYqxymz+kmr7pt59UO+O4U3KSOLHLO0z6RICnzz+l4Xb15h6Hus0mYhDuoMJul3tbT4G4vvLg3GKTb4ZPxeH/qIA=
X-Received: by 2002:a25:cfce:: with SMTP id f197mr518039ybg.51.1569263777347; Mon, 23 Sep 2019 11:36:17 -0700 (PDT)
MIME-Version: 1.0
References: <CAH56bmBmywKg_AxsHnRf97Pfxu4Yjsp_fv_s4S7LXk1voQpV1g@mail.gmail.com> <4D7F4AD313D3FC43A053B309F97543CFA0ADC777@njmtexg4.research.att.com> <LEXPR01MB05607E081CB169E34587EEEF9CA10@LEXPR01MB0560.DEUPRD01.PROD.OUTLOOK.DE> <4D7F4AD313D3FC43A053B309F97543CFA0AF9184@njmtexg5.research.att.com> <CAH56bmC3gDEDF0wypcN2Lu+Ken3E7f_zXf_5yYbJGURBsju22w@mail.gmail.com> <LEJPR01MB11783F8121EC828FF1EC12419C850@LEJPR01MB1178.DEUPRD01.PROD.OUTLOOK.DE>
In-Reply-To: <LEJPR01MB11783F8121EC828FF1EC12419C850@LEJPR01MB1178.DEUPRD01.PROD.OUTLOOK.DE>
From: Matt Mathis <mattmathis@google.com>
Date: Mon, 23 Sep 2019 11:36:05 -0700
Message-ID: <CAH56bmCVErxNnXntSNqxEL2CcEaHuP_oq92OJQJnNDaLCF2T6A@mail.gmail.com>
To: Ruediger.Geib@telekom.de
Cc: "ippm@ietf.org" <ippm@ietf.org>, "MORTON, ALFRED C (AL)" <acm@research.att.com>
Content-Type: multipart/alternative; boundary="00000000000002901805933cb2bf"
Archived-At: <https://mailarchive.ietf.org/arch/msg/ippm/gYBb1lzlM9-SHMnkbq_WCJekqGo>
Subject: Re: [ippm] How should capacity measurement interact with shaping?
X-BeenThere: ippm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF IP Performance Metrics Working Group <ippm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ippm>, <mailto:ippm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ippm/>
List-Post: <mailto:ippm@ietf.org>
List-Help: <mailto:ippm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ippm>, <mailto:ippm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Sep 2019 18:36:23 -0000

This was actually DOCSIS.  I am guessing that the shaping is done at the
CMTS, based on load signals from elsewhere in the network, because the
behaviors vary depending on where the sender is located.   This has all
sorts of interesting implications from a measurement perspective.

For one thing, it is potentially "below the horizon" for many current
techniques.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

We must not tolerate intolerance;
       however our response must be carefully measured:
            too strong would be hypocritical and risks spiraling out of
control;
            too weak risks being mistaken for tacit approval.


On Mon, Sep 23, 2019 at 1:19 AM <Ruediger.Geib@telekom.de> wrote:

> Hi Matt,
>
>
>
> Thanks. My expertise is more with fixed network access. I don’t know how
> access bandwidth is shared between LTE flows at a mobile access.
>
>
>
> On the fixed network side, I’d expect a policy control point to be
> connected to some access line concentrator (be it optical, or copper based
> – coax I can’t tell). To my understanding the configurations having an
> impact at the fixed access are:
>
>    - The access shaper rate (settable)
>    - The access shaper burst size (likely a default value, depending on
>    phys. line card bandwidth)
>    - The queue depth (settable)
>    - AQM parameters (settable to some extent, if deployed)
>
>
>
> A burst which is below or equal to shaper burst size with a temporal
> pacing allowing to the shaper to recover for new tokens will likely see
> line speed at that bottleneck. That should hold in general.
>
>
>
> The shaper kicks in, if more than default shaper burst size packets arrive
> within the given time frame. Then the queue-limit has an impact, if more
> than access shaper rate packets arrive. If the latency added by the
> building queue starts to control the measurement flow rate fast enough,
> loss should be avoided. But then you should be able to measure the shaper
> rate, I’d expect.
>
>
>
> While I doubt that this fits to what you’ve observed, it may still help to
> design a test which either measures available bandwidth or access bandwidth
> in the presence of a shaper:
>
>    - An initial burst may pass a shaper at line speed
>    - A measurement sending up to shaper rate CBR spaced packets which
>    consists of more than “default shaper burst size” packets  should see no
>    added delay and be forwared by shaper rate (it would of course pass with up
>    to “default shaper burst size” packets at that rate too, but then the rate
>    measurement isn’t sustainable, if enough tokens are added between separate
>    measurement streams) .
>    - If the CBR spaced measurement flow sends more than default shaper
>    burst size packets with more than shaper rate, then a queue should build at
>    the shaper.
>
>
>
> You mentioned a token bucket of 1 MB. I’d expect the shaper default burst
> size to depend on the phys. line card bandwidth of the interface, where
> this shaper is configured. I’m not sure, whether to expect a single
> constant value for different implementations. I’d like to describe the
> default shaper bandwidth by a strawman “1ms@phys-interface-bandwidth”
> token bucket. Interfaces may be 1GE, 10GE or 100GE. I’d however not be
> surprised, if different generations of linecards from the same vendor
> support different values for the same physical port bandwidth, and
> different vendors may have different design philosophies.
>
>
>
> Regards,
>
>
>
> Ruediger
>
>
>
>
>
>
>
> *Von:* Matt Mathis <mattmathis@google.com>
> *Gesendet:* Donnerstag, 19. September 2019 23:18
> *An:* ALFRED C (AL) <acm@research.att.com>; Geib, Rüdiger <
> Ruediger.Geib@telekom.de>
> *Cc:* ippm@ietf.org
> *Betreff:* Fwd: How should capacity measurement interact with shaping?
>
>
>
> Ok, moving the thread to IPPM
>
>
>
> Some background, we (Measurement Lab) are testing a new transport (TCP)
> performance measurement tool, based on BBR-TCP.   I'm not ready to talk
> about results yet (well ok, it looks pretty good).    (BTW the BBR
> algorithm just happens to resemble the algorithm described
> in draft-morton-ippm-capcity-metric-method-00.)
>
>
>
> Anyhow we noticed some interesting performance features for number of ISPs
> in the US and Europe and I wanted to get some input for how these cases
> should be treated.
>
>
>
> One data point, a single trace saw ~94.5 Mbit/s for ~4 seconds,
> fluctuating performance ~75 Mb/s for ~1 second and then stable
> performance at ~83Mb/s for the rest of the 10 second test.    If I were to
> guess this is probably a policer (shaper?) with a 1 MB token bucket and
> a ~83Mb/s token rate (these numbers are not corrected for header overheads,
> which actually matter with this tool).  What is weird about it is that
> different ingress interfaces to the ISP (peers or serving locations)
> exhibit different parameters.
>
>
>
> Now the IPPM measurement question:   Is the bulk transport capacity of
> this link ~94.5 Mbit/s or ~83Mb/s?   Justify your answer....?
>
>
>
> Thanks,
>
> --MM--
> The best way to predict the future is to create it.  - Alan Kay
>
> We must not tolerate intolerance;
>
>        however our response must be carefully measured:
>
>             too strong would be hypocritical and risks spiraling out of
> control;
>
>             too weak risks being mistaken for tacit approval.
>
>
>
> *Forwarded Conversation*
> *Subject: How should capacity measurement interact with shaping?*
> ------------------------
>
>
>
> From: *Matt Mathis* <mattmathis@google.com>
> Date: Thu, Aug 15, 2019 at 8:55 AM
> To: MORTON, ALFRED C (AL) <acm@research.att.com>
>
>
>
> We are seeing shapers  with huge bucket sizes, perhaps as larger or larger
> than 100 MB.
>
>
>
> These are prohibitive to test by default, but can have a huge impact in
> some common situations.  E.g. downloading software updates.
>
>
>
> An unconditional pass is not good, because some buckets are small.  What
> counts as large enough to be ok, and what "derating" is ok?
>
>
> Thanks,
>
> --MM--
> The best way to predict the future is to create it.  - Alan Kay
>
> We must not tolerate intolerance;
>
>        however our response must be carefully measured:
>
>             too strong would be hypocritical and risks spiraling out of
> control;
>
>             too weak risks being mistaken for tacit approval.
>
>
>
> ----------
> From: *MORTON, ALFRED C (AL)* <acm@research.att.com>
> Date: Mon, Aug 19, 2019 at 5:08 AM
> To: Matt Mathis <mattmathis@google.com>
> Cc: CIAVATTONE, LEN <lc9892@att.com>, Ruediger.Geib@telekom.de <
> Ruediger.Geib@telekom.de>
>
>
>
> Hi Matt, currently cruising between Crete and Malta,
>
> with about 7 days of vacation remaining – Adding my friend Len.
>
> You know Rüdiger. It appears I’ve forgotten how to typs in 2 weeks
>
> given the number of typos I’ve fixed so far...
>
>
>
> We’ve seen big buffers on a basic DOCSIS cable service (downlink >2 sec)
>
> but,
>
>   we have 1-way delay variation or RTT variation limits
>
>   when searching for the max rate, that don’t many packets
>
>   queue in the buffer
>
>
>
>   we want the status messages that result in rate adjustment to return
>
>  in a reasonable amount of time (50ms + RTT)
>
>
>
>   we usually search for 10 seconds, but if we go back and test with
>
>   a fixed rate, we can see the buffer growing if the rate is too high.
>
>
>
>   There will eventually be a discussion on the thresholds we use
>
>   in the search // load rate control algorithm. The copy of
>
>   Y.1540 I sent you has a simple one, we moved beyond that now
>
>   (see the slides I didn’t get to present at IETF).
>
>
>
>   There is value in having some of this discussion on IPPM-list,
>
>   so we get some **agenda time at IETF-106**
>
>
>
> We measure rate and performance, with some performance limits
>
> built-in.  Pass/Fail is another step, de-rating too (made sense
>
> with MBM “target_rate”).
>
>
>
> Al
>
>
>
> ----------
> From: <Ruediger.Geib@telekom.de>
> Date: Mon, Aug 26, 2019 at 12:05 AM
> To: <acm@research.att.com>
> Cc: <lc9892@att.com>, <mattmathis@google.com>
>
>
>
> Hi Al,
>
>
>
> thanks for keeping me involved. I don’t have a precise answer and doubt,
> that there will be a single universal truth.
>
>
>
> If the aim is only to determine the IP bandwidth of an access, then we
> aren’t interested in filling a buffer. Buffering events may occur, some of
> which are useful and to be expected, whereas others are not desired:
>
>
>
>    - Sender shaping behavior may matter (is traffic at the source CBR or
>    is it bursty)
>    - Random collisions should be tolerated at the access whose bandwidth
>    is to be measured.
>    - Limiting packet drop due to buffer overflow is a design aim or an
>    important part of the algorithm, I think.
>    - Shared media might create bursts. I’m not an expert in the area, but
>    there’s an “is bandwidth available” check in some cases between a central
>    sender using a shared medium and the receivers connected. WiFi and may be
>    other wireless equipment buffers packets also to optimize wireless resource
>    optimization.
>    - It might be an idea to mark some flows by ECN, once there’s a guess
>    on a sending bitrate when to expect no or very little packet drop. Today,
>    this is experimental. CE marks by an ECN capable device should be expected
>    roughly once queuing starts.
>
>
>
> Practically, the set-up should be configurable with commodity hard- and
> software and all metrics should be measurable at the receiver. Burstiness
> of traffic and a distinction between queuing events which are to be
> expected and (undesired) queue build up are the events to be distinguished.
> I hope that can be done with commodity hard- and software. I at least am
> not able to write down a simple metric distinguishing queues to be expected
> from (undesired) queue build up causing congestion. The hard- and software
> to be used should be part of the solution, not part of the problem (bursty
> source traffic and timestamps with insufficient accuracy to detect queues
> are what I’d like to avoid).
>
>
>
> I’d suggest to move discussion to the list.
>
>
>
> Regards,
>
>
>
> Rüdiger
>
>
>
> ----------
> From: *MORTON, ALFRED C (AL)* <acm@research.att.com>
> Date: Thu, Sep 19, 2019 at 7:01 AM
> To: Ruediger.Geib@telekom.de <Ruediger.Geib@telekom.de>
> Cc: CIAVATTONE, LEN <lc9892@att.com>, mattmathis@google.com <
> mattmathis@google.com>
>
>
>
> I’m catching-up with this thread again, but before I reply:
>
>
>
> *** Any objection to moving this discussion to IPPM-list ?? ***
>
>
>
> @Matt – this is a question to you at this point...
>
>
>
> thanks,
>
> Al
>
>
>
> *From:* Ruediger.Geib@telekom.de [mailto:Ruediger.Geib@telekom.de]
> *Sent:* Monday, August 26, 2019 3:05 AM
> *To:* MORTON, ALFRED C (AL) <acm@research.att.com>
> *Cc:* CIAVATTONE, LEN <lc9892@att.com>; mattmathis@google.com
> *Subject:* AW: How should capacity measurement interact with shaping?
>
>
>
> Hi Al,
>
>
>
> thanks for keeping me involved. I don’t have a precise answer and doubt,
> that there will be a single universal truth.
>
>
>
> If the aim is only to determine the IP bandwidth of an access, then we
> aren’t interested in filling a buffer. Buffering events may occur, some of
> which are useful and to be expected, whereas others are not desired:
>
>
>
> -        Sender shaping behavior may matter (is traffic at the source CBR
> or is it bursty)
>
> -        Random collisions should be tolerated at the access whose
> bandwidth is to be measured.
>
> -        Limiting packet drop due to buffer overflow is a design aim or
> an important part of the algorithm, I think.
>
> -        Shared media might create bursts. I’m not an expert in the area,
> but there’s an “is bandwidth available” check in some cases between a
> central sender using a shared medium and the receivers connected. WiFi and
> may be other wireless equipment buffers packets also to optimize wireless
> resource optimization.
>
> -        It might be an idea to mark some flows by ECN, once there’s a
> guess on a sending bitrate when to expect no or very little packet drop.
> Today, this is experimental. CE marks by an ECN capable device should be
> expected roughly once queuing starts.
>
>
>