Re: [tmrg] Follow up from IETF 77 in Anaheim

[Lachlan Andrew <[e-mail]>]

I'm forwarding on behalf of Kevin Mills.  (Thanks, Kevin, for your
contribution.)

Lachlan

-----Original Message-----
From: Mills, Kevin L.
Sent: Wednesday, September 08, 2010 12:06 PM
To: '[e-mail]'
Cc: 'Eddy, Wesley M. (GRC-RCN0)[VZ]'
Subject: Follow up from IETF 77 in Anaheim

In an email conversation subsequent to the Anaheim IETF meeting,
Lachlan Andrew encouraged NIST to share a technical description of the
methods we used to model and analyze global behavior when comparing
various proposed congestion control mechanisms for the Internet.
(For more information on this study see
http://www.nist.gov/itl/antd/Congestion_Control_Study.cfm.)

This email provides a partial response to Lachlan's suggestion. We
have produced three (related) technical papers, attached to this
email.

NIST-904682 "How to Model a TCP/IP Network using only 20 Parameters"
describes the reduced-parameter simulation model that we constructed
to use for our study. Using a reduced-parameter model enabled us to
reduce the computation requirements for our experiments. (A slightly
shorter version of this paper will be presented at the 2010 Winter
Simulation Conference.)

NIST-904961 "An Efficient Sensitivity Analysis Method for Network
Simulation Models" describes our method to identify those parameters
(from among the 20 in our model) that produced the most significant
changes in model responses. We could base the experiments in our study
on this subset of parameters, which further reduced the computational
requirements needed to conduct our simulations. In addition, the
method outlined in this paper was repeated in each of our experiments,
where we exercised each congestion control mechanism under 32
conditions that were spread throughput the parameter search space in
an orthogonal and balanced form. (We have proposed a case study based
on this method for presentation at the 2010 Winter Simulation
Conference.)

NIST-906588 "Comparison of Two Dimension-Reduction Methods for Network
Simulation Models" discusses two different approaches we used to
reduce the response space in our experiments. In this particular paper
we reduced a 22 dimensional response space to 4 (using one method) and
7 (using a second method). The goal of response space reduction is to
identify the most important aspects of our model's behavior, which
allows us to represent those aspects in a complete and balanced way
within our experiments. (We have proposed a case study based on this
method for presentation at the 2010 Winter Simulation Conference.)

These three papers give a foundation for the methods we used in our
study. What these papers do not cover are various, additional data
analysis techniques that we used. Should there be sufficient interest,
then we could write a 4th paper describing these additional analysis
methods. All of our data analysis methods relied on the orthogonal and
balanced experiment design techniques described in NIST-904961.

The main power of the methods we used arises from 2-level, orthogonal
fractional factorial (OFF) experiment designs (NIST-904961), which
allow us to vary all experiment parameters simultaneously while
probing the search space widely in a balanced fashion. This leads to
broader insights for lower computational cost than the one
factor-at-a-time (FAT) experiment designs that are typically adopted
by network simulation practitioners. In addition, the broader insights
provided by OFF experiment designs can reveal areas that might be
profitable to study with FAT designs.

While we used the methods described here to study some specific
Internet congestion control mechanisms, the methods are general and
can be applied to most experiments that require simulating a search
space that is infeasible to compute. In fact, we are currently using
these same methods to investigate resource allocation algorithms that
might be used in Infrastructure as a Service (IaaS) computational
clouds.

Kevin

NIST
100 Bureau Drive Stop 8920
Bldg. 222/B218
Gaithersburg, MD 20899-8920

Tel: [phone]
Fax: [phone]
email: [e-mail]
web: http://w3.antd.nist.gov/~mills