Re: [aqm] [AQM Evaluation Guidelines]

[Dave Taht <[e-mail]>]

On Wed, Apr 16, 2014 at 8:28 AM, Greg White <[e-mail]> wrote:
> All,
>
> As has been mentioned, CableLabs work on AQM evaluations has been (so far)
> entirely devoted to evaluating performance for upstream AQM algorithms that
> would be implemented in a residential cable modem.   While the traffic is,
> of course, bi-directional, the focus was on identifying realistic test cases
> that would provide insight on the performance of the algorithms that we were
> considering (in both an absolute and a relative sense).  For evaluating
> downstream AQM performance I would definitely add some additional traffic
> types, and video traffic would be at the front of my mind there.  In order
> to do that justice, I think you'd want to have a HAS model that incorporates
> adaptive bitrate capability (but what algorithm?) and estimates QoE using
> something like the de Vriendt model.

+1

> For progressive download, it would
> seem that the salient metrics would be initial buffering delay and then
> something related to the number and duration of rebuffering events.
>
> I agree with others that reducing the web model to a single 700kB file
> download is unrealistic.  Also, for recent work we've increased the object
> sizes to correspond to a projected 2018 page size (extrapolating from
> http://httparchive.org/trends.php) of about 7MB.

While the trendline is impressively "up":

http://httparchive.org/trends.php?s=All&minlabel=Nov+15+2010&maxlabel=Apr+1+2014

I'm reluctant to do a linear extrapolation of that trendline from 3
years of data. Perhaps
a longer term store, like that of archive.org, could be explored. I'm
not one to predict
a "Peak Web", but the idea of a typical web page load cracking 7MB in
5 years disturbs
me.

>
> For our 2018 model of Internet traffic (again, upstream focused) where the

I do like, very much, trying to come up with projections for what traffic will
look like in 5 years. I'll ask around...

> user's broadband speed is projected to be 1Gbps down and 200Mbps up, we're
> using the following 9 traffic loads:
>
> N F1 F2 Fs    W VG C T
> 1 0  0  -     1 1  0 0
> 2 1  1  inf   1 1  0 0
> 3 3  3  50MB  1 1  0 0
> 4 3  3  50MB  1 1  6 0
> 5 1  1  19MB* 1 1  0 0
> 6 3  3  50MB  4 4  0 0
> 7 3  3  50MB  4 4  6 0
> 8 5  5  2.5MB 4 4  6 0
> 9 3  3  50MB  4 1  0 100
>
> where:
> N: traffic load index
> F1: number of simultaneous FTP uploads with 20ms RTT
> F2: number of simultaneous FTP uploads with 100ms RTT

I note that the average RTT in the US is apparently 38ms,
and trending downward at about 1ms/year on wired connections.

Key web sites in major cities (e.g. googles of the world) are
below 5ms. I wish I knew netflix's typical RTT from ISP
co-location facilities... high bandwidth needing providers
will continue to also reduce their RTT's as much as they can.

> Fs: FTP filesize
> W: number of simultaneous web users

I have two mental models for cable internet connectivity - one is
in the home, the other is in small (and increasingly larger) business.

Given the first concept, it is hard to extrapolate as far as your
larger models, and given the second, I'd imagine web, voip,
and videoconferencing traffic to be a much larger percentage
of the overall model.

> VG: number of simultaneous VoIP/gaming sessions
> C: CBR data rate (Mbps)

We really need a good model for videconferencing.

I have no idea (except for live streaming delivery) what
the CBR stream is intended to represent. Security cameras backhauled to
Big Brother?

Another item is the amount of "noise" traffic - dns, iot devices reporting
tempurature, weather, etc, being transported.

> T: number of torrent (LEDBAT) connections

Torrent does not behave as per your model; I just had a long meeting
with brahm over that. Presently per "new" torrent you typically see dozens of
sessions "connected", but only 6 or so sessions exchanging data in
both directions,
with a new connection explored and rotated to every 15 seconds or so.

After a file is fully transferred, the same logic remains, but upload only.
Anticipating that a given site would have 16 simultaneous torrents over
a 100 active connections being redistributed seems overlarge, and I imagine
that a typical number is bound much lower by the bittorrent software.

> *Filesize and repetition pattern chosen to exercise DOCSIS "powerboost"
> feature

Yes, I'd like not only to have that but to actually have running code in
linux that could emulate it... :)

>
>
> Since evaluation comes before deployment (by a long lead time in some
> cases), and, once deployed, algorithms may have a long life, I would
> encourage the AQM Evaluation Guidelines draft to consider not just what
> traffic looks like historically, but what we can reasonably predict for the
> future.

+1

Now that we are transporting hi resolution video, voice, web, and data
in addition to the traditional email and chat, I have great difficulty assuming
that bandwidth demands will continue to grow as they have, as the new stuff
is mostly "fixed" in its bandwidth needs, rather than dynamic like tcp.

... Unless someone comes up with a way of encapsulating other senses or
some application that truly demands high bandwidth that can work at
multiple RTTs.

There is a lot downward pressure to "do more with less", for example,
as with things like webrtc that are bundled in the browser and don't
require a lot of javascript support, and a lot of caching of core code
that takes place, and a lot of work in reducing RTTs in new protocols
like QUIC.

(but I freely admit I could be wrong and would love to work on projections
 that make sense!)

> -Greg
>
>
> From: <Akhtar>, "Shahid (Shahid)" <[e-mail]>
> Date: Tuesday, April 15, 2014 at 4:47 PM
> To: Nicolas KUHN <[e-mail]>
> Cc: Richard Scheffenegger <[e-mail]>, "[e-mail]" <[e-mail]>
>
> Subject: Re: [aqm] [AQM Evaluation Guidelines]
>
> Hi Nicholas,
>
> Please see comments inline.
>
> Best regards,
>
> -Shahid.
>
> ________________________________
> From: Nicolas KUHN [mailto:[e-mail]]
> Sent: Tuesday, April 15, 2014 3:24 AM
> To: Akhtar, Shahid (Shahid)
> Cc: [e-mail]; Richard Scheffenegger
> Subject: Re: [aqm] [AQM Evaluation Guidelines]
>
> Hi Shahid Akhtar, all,
>
> After the discussions in London IETF 89, there has been lots of
> modifications to the draft organisation (as we already advertised on the
> list).
> The authors and I just discussed your points considering the size of the
> files for the bulk transfer and the HTTP traffic.
> In the current version of the draft, we consider the values mentioned in the
> cablelabs work [0].
>
> As a result, we consider bulk TCP transfers (repeating 5MB file
> transmission) and realistic HTTP web traffic (repeated download of 700kB).
> As a reminder, please find here the comments of Shahid Akhtar regarding
> these values:
>
> "Bulk TCP transfer: (continuous file transmission, or repeating 5MB file
> transmission);"
> The largest type of traffic on the Internet that uses bulk TCP - or large
> files via TCP is progressive download. The average speed of a Youtube video
> is over 3Mbps and it is on average 4.5 minutes long - so the average file
> size is considerably larger than 5MB. We have used an average of 38MB
> assuming 480p video and 4.5 minutes for some of our experiments. The
> inter-arrival time of these files should have a realistic random
> distribution as well as size of the files.
>
> SA: The Cablelabs document as mentioned below has a set of tests for Bulk
> TCP for uploading traffic, they use a single continuous TCP flow for certain
> testcases and multiple TCP flows with 20MB or 250KB files repeatedly being
> sent upstream for other testcases. Their application is upstream traffic
> which behaves differently from downstream traffic. A Sandvine document on
> Internet traffic
> (https://www.sandvine.com/downloads/general/global-internet-phenomena/2013/exposing-the-technical-and-commercial-factors-underlying-internet-quality-of-experience.pdf)
> shows that real-time entertainment (Netflix, YouTube and others) constitute
> 68% of downstream traffic, (Figure 9) of which YouTube (progessive download)
> is at 17% of total traffic. Using repeated transfers of TCP files as
> suggsted in the Cablelabs work would miss the effect of the random on-off
> patterns that PD flows generate on the downstream portion. So I would still
> recommend the use of PD (YouTube) statistics such as above for modeling the
> Bulk TCP traffic downstream traffic.
>
> "Realistic HTTP web traffic (repeated download of 700kB);"
> According to statistics that Google collected from 2010 about web traffic,
> the average GET file size for web traffic was only 7KB. It is very important
> to have the size of these files correct since AQMs can help transfers of
> small files when large file downloads are also occurring. The effect of AQM
> on a 7KB file would be very different than on a 700KB file. Web traffic also
> has a complex inter-arrival time relationship between the GET requests as
> well as the pages that produce these GET requests. However to make the
> testing simpler, a repeated download of a 7KB file with a Pareto distributed
> size and Pareto distributed inter-arrival time may be an OK compromise.
>
> SA: I believe Toke has already responded that the CableLabs work mentioned
> below refers to another document, which uses 25KB for responses to web page
> GETs. This is hugely better than using repeated 700KB downloads and I am OK
> with using the model in the other document. I am still concerned that the
> average size of the reponse maybe less than 25KB, The statistics collected
> by Google (https://developers.google.com/speed/articles/web-metrics) show an
> average size of 7KB, but they are 4 years old and I cannot find a more
> recent number.
>
> We expect to stick with the cable labs values [0]. Please let us know if you
> have any comments regarding this choice.
>
> The Cablelabs work does not mention any HTTP adaptive streaming (HAS)
> traffic models - would that be included in the guidelines?
>
> Kind regards,
>
> Nicolas KUHN
>
> [0]
> http://www.cablelabs.com/wp-content/uploads/2013/11/Active_Queue_Management_Algorithms_DOCSIS_3_0.pdf
>
> On Feb 20, 2014, at 6:48 AM, "Akhtar, Shahid (Shahid)"
> <[e-mail]> wrote:
>
> Hi Nicholas and other authors,
>
> Thank you for producing this document. I had some (long) comments where I
> have also suggested some solutions/text, hopefully you find them helpful.
> When quoting from the document, I have used quotes and I have listed my
> comments as 1-4.
>
>
> 1. The number of tests recommended per AQM scheme seems to be quite large as
> well as many tests that may be best done together are done independently.
>
> For example, there are different tests for fairness between flows (section
> 3.2.3), level of traffic (mild, medium, heavy) (section 3.2.4), different
> types of bursty traffic (3.2.2) and different network environments (section
> 3.3). In order to obtain a realistic estimate of AQM comparison, many of
> these effects may need to be tested together. For example to increase level
> of traffic (in section 3.2.4) only long term TCP flows of 50s are proposed.
> It would be better to test the level of congestion with realistic traffic
> (and with realistic proportions) as proposed in section 3.2.4. Another
> example would be to test fairness between flows of same type - e.g. long
> term TCP flows when multiple types of TCP flows are present at a AQM
> controlled buffer or test fairness between flows of different types - e.g.
> long term TCP flows and web flows.
>
> I noticed that in section 3.3 different network environments are mentioned.
> I suggest that tests with appropriate realistic traffic be done on each of
> these enviroments with various levels of congestion. One could measure
> fairness from these tests without the need for additional tests. This may
> have the benefit of reducing overall number of tests as well.
>
>
> 2. There are four network enviroments mentioned in the document (section
> 3.3):
> Wifi - in home network.
> Data-Center communications.
> BB wireless/satellite.
> Low and high buffers
>
> This is a good set, however, recommed to add:
>
> Access networks - and further split access networks into DSL/FTTH networks
> and Cable access networks. Cable access networks may have much larger number
> of flows than DSL/FTTH since they are shared by a large number of customers
>
> There is another type of network environment - narrowband wireless (mobile)
> networks where the AQM could be placed on the eNodeB, but perhaps this is a
> complicated scenario, and best left for later or in another document.
>
>
> 3. I noticed that the modeling of different types of realistic traffic could
> be made more accurate (section 3.2.2)
>
> "Bulk TCP transfer: (continuous file transmission, or repeating 5MB file
> transmission);"
> The largest type of traffic on the Internet that uses bulk TCP - or large
> files via TCP is progressive download. The average speed of a Youtube video
> is over 3Mbps and it is on average 4.5 minutes long - so the average file
> size is considerably larger than 5MB. We have used an average of 38MB
> assuming 480p video and 4.5 minutes for some of our experiments. The
> inter-arrival time of these files should have a realistic random
> distribution as well as size of the files.
>
> "Realistic HTTP web traffic (repeated download of 700kB);"
> According to statistics that Google collected from 2010 about web traffic,
> the average GET file size for web traffic was only 7KB. It is very important
> to have the size of these files correct since AQMs can help transfers of
> small files when large file downloads are also occurring. The effect of AQM
> on a 7KB file would be very different than on a 700KB file. Web traffic also
> has a complex inter-arrival time relationship between the GET requests as
> well as the pages that produce these GET requests. However to make the
> testing simpler, a repeated download of a 7KB file with a Pareto distributed
> size and Pareto distributed inter-arrival time may be an OK compromise.
>
> I noticed that adaptive streaming is not modelled as a type of realistic
> traffic. It is probably the most important and largest flow on the Internet
> today - if we look at a recent Sandvine report of Internet traffic, about
> 75% of Internet traffic is real-time entertainment of which the largest
> component is adaptive streaming based video traffic. In order to simulate
> adaptive streaming traffic, one would need a client that changes video
> quality rates. To keep testing simpler, an option may be to generate
> realistic chunk sized files at regular intervals. Average netflix speeds are
> around 2Mbps. Typical chunk sizes range from 2s to 10s, but 4s may be a good
> average. The chunk sizes should be varied with some random distribution as
> is typically observed from real codecs.
>
>
> 4. In section 2.2.5, QoE metrics are mentioned. It may be valuable to spell
> out some standard models or expressions to derive QoE from network
> parameters so that these QoE can be compared between AQMs. In the literature
> there is work that derives closed form expressions for QoE. The following
> are examples and suggestions:
>
> Video - the work by Johan de Vriendt et al , "Model for estimating QoE of
> Video delivered using HTTP Adaptive Streaming", IFIP/IEEE Workshop on QoE
> Centric Management, May 2013 may be useful here. The authors used over 500
> samples to derive their a closed form expressions. The authors derive the
> MOS = alpha*(average bit rate) + beta*(standard deviation of bit rates) +
> sigma*(no of bit rate switches) + gamma and then produce good values for
> alpha, beta, sigma and gamma.
>
> Voice - The work in "Voice quality prediction models and their application
> in VoIP networks" by L. Sun et al has closed form expressions for MOS scores
> for most commercial codes. They derive a long closed form expression with
> parameters a-j for key codecs (and then provide a-j for most commercial
> codecs) based on end-to-end and packet loss.
>
> Web - The delay for one average web GET file could be used as the
> representative for the page load time
>
> Key metric for AQMs on access networks may be QoE since it provides and
> end-user view. We have seen that often the absolute goodput does not matter
> as much as the stability of the flow for Video (which tends to be the
> dominant traffic). Larger goodput may provide higher VQ levels for Video,
> but if higher volatility is also produced then that higher VQ level may not
> produce a higher QoE.
>
> On the other hand, best metrics inter data center traffic may be goodput and
> delay since most of that traffic is likely to be inter machine traffic.
>
> Best regards,
>
> Shahid Akhtar.
>
> -----Original Message-----
> From: aqm [mailto:[e-mail]] On Behalf Of Nicolas KUHN
> Sent: Friday, February 07, 2014 5:58 AM
> To: Richard Scheffenegger
> Cc: [e-mail]
> Subject: [aqm] [AQM Evaluation Guidelines]
>
> Dear all,
>
> On the behalf of the contributors to the AQM Evaluation Guidelines, I
> encourage active discussion on the draft that we have written.
> I just submitted the draft as an individual draft to the I-D Submission Tool
> [0].
>
> Please let us know what you think of the current document.
>
> Kind regards,
>
> Nicolas KUHN
>
> [0] http://datatracker.ietf.org/doc/draft-kuhn-aqm-eval-guidelines/
>
>
> On Feb 5, 2014, at 10:01 PM, "Scheffenegger, Richard" <[e-mail]> wrote:
>
> Hi,
>
> a new month, a new status report.
>
> First of all, Wes and I as chairs would like to thank the editors who have
> stepped forward to work on the AQM Evaluation Guideline draft. We are really
> thankful for their burst of efforts in the last couple weeks!
>
> We expect that that a document will be ready for submission into the I-D
> repository well before cutoff, as an individual draft, so that the WG can
> see what the state of thinking is currently. Also, once the document is
> published on datatracker we'd like to encourage active discussion on it.
>
> If the submitted -00 draft is felt to have a fairly complete outline of what
> the current thinking in the WG is, we may be able to ask the WG for formal
> adoption during this IETF meeting, or shortly thereafter.
>
>
>
>
> - WG Milestones:
>  - Submit AQM recommendations to IESG for publication, obsoleting RFC
> 2309 (Goal: January 2014)
>    - draft-ietf-aqm-recommendation is accepted towards this milestone
>    - http://datatracker.ietf.org/doc/draft-ietf-aqm-recommendation/
>    - the draft has been updated per comments received
>    - if the authors are comfortable, a WGLC might be made on the next
> revision
>    - we would like to hear from other authors of RFC 2309 on this
> document, if anyone has contacts to them.
>
>
>  - Submit AQM algorithm evaluation guidelines to IESG for publication
> as Informational (Goal: July 2014)
>    - An editor team has come forth and is working on this
>    - A draft should be available for discussion in the London IETF
>    - We encourage discussion on the list and during the meeting, if this
>       draft should be adopted by the working group.
>
>  - Submit first algorithm specification to IESG for publication as
> Proposed Standard (Goal: December 2014)
>    - Since any Proposed Standard algorithm should be in line with the
> recommendations and be passable versus the evaluation guidelines, this
> milestone is dependend on the progress of the two work items above.
>    - Currently the only algorithm spec with a complete and active
> individual-submission draft is PIE
>
> - Other items:
>  - draft-pan-aqm-pie is under active work as a proposed algorithm:
>    http://tools.ietf.org/id/draft-pan-aqm-pie-00.txt, however the draft
>    has expired and should be refreshed.
>  - draft-nichols-tsvwg-codel is expired; Dave Taht or others may
> revive it and/or describe pairing with FQ/SFQ algorithms:
>    http://tools.ietf.org/id/draft-nichols-tsvwg-codel-01.txt
>  - Other algorithm specifications are welcome!
>    - Though, we are not planning on adopting algorithms until
> recommendations and evaluation guidelines are mostly stable
>
>
> Richard Scheffenegger
>
> _______________________________________________
> aqm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/aqm
>
>
> _______________________________________________
> aqm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/aqm
>
>
>
> _______________________________________________
> aqm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/aqm
>



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article