Re: [tsvwg] Switch testing at 25G with ECN --SCE Draft

["Scaglione, Giuseppe" <[e-mail]>]

Greg,

>> Just to be super clear, this isn't a hardware implementation of SCE running at 25Gbps.

I am not sure I follow. The Test setup section of the paper clearly describes the hardware used -- severs, switch, cables. 
What I cannot disclose at this point is the exact model and characteristics of the HPE Aruba Switch used. Yet, it is a "real" Ethernet Switch, providing 25Gbps connectivity, configured to do bridging across the four ports and implementing RFC3168 with the ECN remarking configuration described in my previous email and on the paper.

Cheers,
Giuseppe 

-----Original Message-----
From: Greg White [mailto:[e-mail]] 
Sent: Friday, August 9, 2019 12:17 PM
To: Scaglione, Giuseppe <[e-mail]>; [e-mail]
Cc: [e-mail]
Subject: Re: [tsvwg] Switch testing at 25G with ECN --SCE Draft

Rodney and Giuseppe, thanks for the clarification.

Just to be super clear, this isn't a hardware implementation of SCE running at 25Gbps.   It is essentially using a 25Gbps DCTCP switch implementation (with an adjustable marking probability slope) to study the behavior of the two experimental SCE congestion controllers in a data center environment.   If this is considered to be a scenario for deployment of SCE congestion controllers, then I would again suggest running the experiment also with the stock DCTCP congestion controller for comparison, since that is what you'd be trying to displace.

-Greg


On 8/8/19, 6:00 PM, "Scaglione, Giuseppe" <[e-mail]> wrote:

    Greetings,
    
    Replying to Greg:
    
    >>If I'm reading this correctly, the marking implementation in the switch is actually DCTCP-style (marking CE
    >>based on instantaneous queuing delay) in a single queue, rather than SCE-style.   
    
    The switch only knows RFC 3168 rules. That is, if packet is IP AND header is marked ECN capable (10 | 01) the packet can be remarked. The remarking probability is given by the formula on page 5:
    
    remark_probability = probability_rise *(current_mem_utilization -mem_start_point) ( all % values)
    
    In the paper, anywhere it says SCE testing the settings were:
    probability_rise = 1
    mem_start_point = 0
    Current_mem_utilization --> this is dynamic, goes from 0-100% provided by the HW queue.
    
    Therefore, for SCE testing the switch was remarking purely function of the queue memory utilization:
    
    remark_probability =  current_mem_utilization  
    
    Which produces that 'early warning' of "some congestion" which is mentioned in the sce-draft, and would never occur in traditional switch setting for CE remarking. 
    Yes -- all testing was using 1 of the 8 egress queues of the switch ports. It would be easy to do a multi queue one.
    
    >>Could you provide some information on the buffer size?  It looks like it may be ~200us
    
    Sorry cannot share any details on the HW and its memory at this point.  
    
    >>How does this marking approach compare to existing DCTCP deployments (i.e. do they generally use a 
    >>hard marking threshold or do they similarly use a ramp)?
    
    In general, some hardware implementation will start remarking at given queue depth (hard start) and then they apply a probability curve more or less aggressive. Other hardware (like the one in the experiment) gets continuous feedback on the queue depth and then applies the probability curve function of such memory usage. 
    Looking at the equation above would be interesting to try to play with the probability_rise as well, which was always left at 1.
    
    >>> Also, do you have an explanation for Pic 1, where the ping time when the link is idle is almost double the 
    >>ping time when the link is saturated?
    
    >>I do have one for this, when the link is idle your receiver is not going to see the packet until a device 
    >>interrupt occurs, which may be a good deal of time as these are not simple single packet/single interrupt 
    >>type nics.  Basically when idle the ping packets are having to wait until either the poll threshold of the 
    >>driver occurs, or something else causes the driver to interrupt.  At high packet rates your delivering a near 
    >> continuous stream of interrupts to the CPU and thus your pings get processed by the receiver almost 
    >>instantly.
    
    I agree with Rodney. I am not sure if the Kernel is using NAPI and doing Interrupt driven Packet Processing and then switching to polled -- if just polled the explanation makes perfect sense.
    
    Best Regards!
    Giuseppe
    
    
    -----Original Message-----
    From: Rodney W. Grimes [mailto:[e-mail]] 
    Sent: Thursday, August 8, 2019 4:33 PM
    To: Greg White <[e-mail]>
    Cc: Scaglione, Giuseppe <[e-mail]>; [e-mail]
    Subject: Re: [tsvwg] Switch testing at 25G with ECN --SCE Draft
    
    > Hi Giuseppe,
    > 
    > Thanks for sharing your work.   At the TSVWG meeting at IETF105 it was mentioned that there was a hardware implementation of SCE running in a 25Gbps switch (1:47:30 in https://www.ietf.org/audio/ietf105/ietf105-placeducanada-20190726-1220.mp3  ).  I presume this is the implementation that was being referred to.
    
    As the person that made that comment I'll confirm that Yes, this is the implementation I was refering to.
    
    > If I?m reading this correctly, the marking implementation in the switch is actually DCTCP-style (marking CE based on instantaneous queuing delay) in a single queue, rather than SCE-style.   The CE marks are then transformed into SCE marking by the receiver itself before being handed to dctcp-sce or reno-sce.  So, in essence this test appears to be a study of various AQM ramp options and modified ECN congestion responses in a homogeneous DCTCP datacenter environment, rather than being an SCE experiment specifically.
    
    I'll defer to Giuseppe for details on that, but I do not believe this is simple dctcp style ce threshold marking when they are running in sce test mode.  See figure 1 and associated text.
    
    As a person somewhat familiar with SCE I can state that dctcp ce marking as used in a data center and the end node response to that ce mark is infact basically the same as SCE, ie there is no latching of CE state until CWR at the receiver, and the marks are in proportion to the queue depth.  BUT by using a different ECT() marking and a different TCP feedback bit you can do both CE and SCE at the same time, which the classical data center dctcp modifications do not allow for.
    
    > 
    > Could you provide some information on the buffer size?  It looks like it may be ~200us.   How does this marking approach compare to existing DCTCP deployments (i.e. do they generally use a hard marking threshold or do they similarly use a ramp)?  Have you thought about adjusting the ramp so that 100% marking occurs before you reach the tail drop limit?  I wonder if that would eliminate the packet drops.
    > 
    > For comparison, you may as well use stock DCTCP which would avoid the receiver needing to translate the CE marks into SCE marks, since DCTCP can just interpret the CE marks directly.   But, I didn?t think that the goal of SCE was to replace DCTCP in datacenters anyway?
    
    The goal of SCE is to provide a robust congestion mechanism, without having to be restricted to use in a datacenter.
    
    > 
    > Are the tables missing some data?  I don?t see the bandwidth/retries/packet drop data for the continuous TCP connections.
    > 
    > Also, do you have an explanation for Pic 1, where the ping time when the link is idle is almost double the ping time when the link is saturated?
    
    I do have one for this, when the link is idle your receiver is not going to see the packet until a device interrupt occurs, which may be a good deal of time as these are not simple single packet/single interrupt type nics.  Basically when idle the ping packets are having to wait until either the poll threshold of the driver occurs, or something else causes the driver to interrupt.  At high packet rates your delivering a near continuous stream of interrupts to the CPU and thus your pings get processed by the receiver almost instantly.
    
    > 
    > Best Regards,
    > Greg
    > 
    > 
    > 
    > From: tsvwg <[e-mail]> on behalf of "Scaglione, Giuseppe" <[e-mail]>
    > Date: Tuesday, August 6, 2019 at 3:36 PM
    > To: "[e-mail]" <[e-mail]>
    > Subject: [tsvwg] Switch testing at 25G with ECN --SCE Draft
    > 
    > Greetings,
    > 
    > The attached document contains data collected testing with an Ethernet switch and the new TCP-dctcp-sce and TCP-reno-sce algorithms, draft-morton-taht-tsvwg-sce-00.
    > 
    > Best regards,
    > 
    > 
    > Giuseppe Scaglione
    > Distinguished Technologist
    > R&D Lab Hardware
    > Aruba Networks
    > Hewlett Packard Enterprise
    > 8000 Foothills Blvd.  (MS 5555)
    > Roseville, CA   95747
    > Tel : [phone]
    > [cid:image001.jpg@01D11639.32250950][cid:[e-mail]78900]
    > 
    > 
    Content-Description: image001.jpg
    
    [ image/jpeg is not supported, skipping... ]
    
    Content-Description: image002.png
    
    [ image/png is not supported, skipping... ]
    
    -- 
    Rod Grimes                                                 [e-mail]