Re: Comments on your SCPS-TP testing report

William Ivancic <wivancic@grc.nasa.gov> Mon, 23 September 2002 13:13 UTC

Message-Id: <4.2.1.20020923075154.00d50bc0@popserve.grc.nasa.gov>
X-Sender: caivanc@popserve.grc.nasa.gov
X-Mailer: QUALCOMM Windows Eudora Pro Version 4.2.1
Date: Mon, 23 Sep 2002 09:13:36 -0400
To: "Adrian J. Hooke" <adrian.j.hooke@jpl.nasa.gov>
From: William Ivancic <wivancic@grc.nasa.gov>
Subject: Re: Comments on your SCPS-TP testing report
Cc: tcpsat@grc.nasa.gov, Diepchi.T.Tran@grc.nasa.gov, Frances.J.Lawas-Grodek@grc.nasa.gov, Robert.P.Dimond@grc.nasa.gov
In-Reply-To: <5.1.0.14.2.20020920110038.0261f828@mail1.jpl.nasa.gov>
Mime-Version: 1.0
Content-Type: multipart/alternative; boundary="=====================_5423240==_.ALT"
Sender: owner-tcpsat@grc.nasa.gov
Precedence: bulk
Status: RO
Content-Length: 21426
Lines: 450

Adrian,

Thanks for the input.  It has been received in sufficient time to get into 
the final report.

We will consider you input.  I do have some questions, however, regarding 
some of the suggestions.

We tested and set the optimum receiver buffer size starting a twice the 
Bandwidth-Delay product (BDP) and found for our test setup with the OS used 
(which appeared to be the most stable OS) that the optimal setting was 
slightly less the 1 BDP.  So, we could say that "in theory" the receive 
buffer should be 2 x BDP, but the data did not support that of Solaris.

IMHO, protocol tuning doesn't scale well - particularly in non-gateway 
implementations and for systems such as NASA's planned sensorWeb.   I would 
really like to see better auto-tuning features developed.

The paper on performance for "SCPS-TP assumed corruption", is for a gateway 
implementations.  All data is received at the gateway and buffered.  Thus, 
all data is buffered at one source and ECN type mechanisms can be 
implemented on the point-to-point link.  This is fine, but is a very 
limited case.  I would not be comfortable extrapolating such performance to 
general deployments of "SCPS-TP assumed corruption" over fully meshed 
networks.  Am I missing something?  Gateway testing is great, but assuming 
everything will use a gateway is ..... maybe not to good.   If one encrypts 
prior to the gateway (very common in a military environment), your limited 
as to what you can do.  I'm always a bit fearful when I see testing 
performed using gateways in that I fear decision makers may not understand 
that those results may only be valid when used in an architecture that 
allows for deployment of a gateway.


Assuming ECN in a fully meshed network may be wishful thinking.  Although 
ECN concepts have been around and experimentally show to be advantageous in 
an ideal world,  I believe, to date, deployment of ECN is limited at 
best.   I'm  may be incorrect here as I do not have the compete  knowledge 
of what is deployed in the backbone.  A quick breeze through the QoS 
commands on Cisco routers did not show ECN settings, but that was a very 
quick search.

Best Regards,


Will



At 11:07 AM 9/20/02 -0700, Adrian J. Hooke wrote:
>Hi William:
>
>The SCPS team has been alerted that you are asking for comments on your 
>SCPS-TP testing report by 21 September:
>
>http://roland.grc.nasa.gov/~ivancic/papers_presentations/SCPS_TCP_RATE_FINA 
>L_0903.pdf
>
>Unfortunately we don't have the resources to do an in-depth technical 
>evaluation of your work. However, we have taken a quick glance at the 
>results and have two recommended changes and some comments. In particular, 
>we would strongly suggest that (for consistency with the rest of the 
>technical data) you should re-write the Abstract, Executive Summary 
>(section 1) and Conclusions (Section 11) along the lines suggested below.
>
>Best regards
>Adrian Hooke, NASA-JPL
>
>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>RECOMMENDED CHANGE:
>
>1 ABSTRACT
>
>Tests were performed at Glenn Research Center (GRC) to compare the 
>performance  of the Space Communications Protocol Standard - Transport 
>Protocol (SCPS-TP,otherwise known as "TCP Tranquillity") relative to other 
>variants of TCP, and to
>determine the implementation maturity level of these protocols - 
>particularly for higher speeds. The testing was performed over reasonably 
>high data rates of up to 100 Mbps with delays that are  characteristic of 
>near-planetary environments. The tests were run for a fixed packet size, 
>but for variously errored environments. This report documents the testing 
>performed to date.
>
>2 EXECUTIVE SUMMARY
>
>There have been numerous  discussions  relative to the role of the SCPS-TP 
>in the  ever-evolving TCP family, particularly with respect to other known 
>TCP features that accommodate long bandwidth-delay products (e.g., large 
>windows, selective acknowledgements). In order to gain a better 
>understanding of the  performance of the SCPS-TP relative to other TCP 
>variants and to determine the maturity of the various  options for use on 
>higher-rate space links, the NASA Space and Data Communications Systems 
>(SCDS) Office requested Glenn Research Center to perform a comprehensive 
>set of tests. Tests were
>performed at Glenn Research Center to validate the operation of the 
>Reference Implementation of the SCPS-TP relative to its controlling CCSDS 
>specification, and to perform a comprehensive comparison of SCPS-TP 
>protocol options relative to other TCP options.
>
>We have studied the effect of delay and BER on the performance of 
>congestion-friendly and rate-based protocols in emulated space links, both 
>under uncongested conditions and with limited congestion. The results 
>correlate well with previous testing of TCP options, including the SCPS-TP:
>    * The single stream and multi-stream test results clearly illustrate 
> that the TCP-Vegas enhancements to TCP (that are implemented by the 
> SCPS-TP) provide measurable performance improvements over a TCP SACK 
> implementation tested in  high bandwidth-delay product environments. 
> Since performance considerations are subjective, the operational value of 
> these performance increases can best be assessed by the users of specific 
> applications hosted within those environments.
>    * Very small transactions such as command and control will probably 
> see little difference in performance for any variant of TCP.
>    * In extremely error-prone environments with high RTT latencies, use 
> of a rate-based TCP variant such as that provided by SCPS-TP is 
> advisable, assuming that the network implementation is properly 
> engineered. Users are cautioned  against using rate-based protocols on 
> networks where resource allocation policies are based on contention.
>    * The tested SCPS-TP implementation was the protocol Reference 
> Implementation, which exists as a user application rather than as a more 
> generalized protocol service provider. For some deployments, an in-kernel 
> protocol service provider may prove more desirable than the deployment of 
> application level protocol service providers. The tradeoff is between the 
> more efficient use of resources and possibly higher performance for 
> in-kernel services versus the ease of maintainability and deployment of 
> application level implementations. Commercial in-kernel and application 
> level implementations of SCPS-TP exist, but have not been tested as part 
> of this effort.
>    * Even with equal performance, the SCPS-TP version of a rate-based 
> protocol may be more desirable to implement than other rate-based 
> protocols (such as the Multicast Dissemination Protocol) since the 
> SCPS-TP is capable of requiring only sending-side modification. This 
> feature of the SCPS-TP eliminates many of the risks associated with 
> requiring universal deployment of new software.
>    * Existing TCP implementations (drawn from a variety of communities, 
> and including the SCPS-TP) appear to satisfy all currently-known space 
> mission needs; however, the space mission community should maintain an 
> awareness of current and future TCP research that is being performed by 
> other communities.
>11 CONCLUSIONS
>
>(Repeat the same changes that are noted above in the Executive Summary.)
>
>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>RECOMMENDED CHANGE:
>
>Sec 4.2.2, "Modified Slow Start", there should be a bullet that reads:
>
>    *  The Vegas Fast Retransmit algorithm is *not* part of SCPS-TP.
>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>COMMENT:
>
>Section 9.1.1.3:  the "optimum receiving buffer size" is mentioned, which 
>is defined to be in a fairly narrow range centered around one Bandwidth 
>Delay Product (BDP) of the long-delay test case.  With SCPS's 
>Vegas-Corruption and Rate configurations, this is not optimal.  Your 
>Solaris implementation might not have shown it due to OS-imposed penalties 
>on outside-the-kernel protocols and buffer management, but in high-error 
>conditions, the SCPS protocols are best configured with greater than one 
>BDP (e.g., two) of storage, so that the time spent waiting for a 
>retransmission to be received does not cause the sender to become window 
>limited.  This isn't typically an issue with native TCP, as the error 
>recovery regime in the congestion control mechanisms limits the protocol's 
>ability to consume buffer space.
>
>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>COMMENT:
>
>You appear to have missed the most important "Multiple Flow" test to 
>conduct.  In Section 9.3, you state that "In addition, 
>SCPS-Vegas-Corruption was not tested in this environment, as it would have 
>been a misapplication of the protocol."  This is seriously incorrect.  The 
>attached MILCOM paper specifically examines the issue of use of the 
>Vegas-Corruption configuration for congestion control in the presence of 
>errors.  It's absolutely the *RIGHT* application of the SCPS-TP protocol, 
>and is a significant omission from your paper.
>
>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>COMMENT:
>
>Also, in that same section 9.3, you seem to have missed the opportunity to 
>draw some relatively important conclusions:
>
>a) SCPS Vegas Congestion proved to be *more fair* than TCP with the Van 
>Jacobson congestion control algorithm.
>
>b) Because there were fewer retransmissions required, the SCPS Vegas 
>algorithm was also *more efficient* than TCP with the Van Jacobson 
>congestion control algorithm.
>
>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>COMMENT:
>
>In Section 10, you cite some problems with TCP Vegas in general (the last 
>three bullets before Section 11).  It should be noted that the issue cited 
>in the second bullet is easily addressed and anticipates the deployment of 
>Explicit Congestion Notification (ECN).  The other two bullets cite issues 
>in which Vegas algorithms are less aggressive than the Van Jacobson 
>algorithm, and will, no doubt, receive attention should these become real 
>problems.
>
>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>-end-
>