Re: [tsvwg] Switch testing at 25G with ECN --SCE Draft

"Scaglione, Giuseppe" <giuseppe.scaglione@hpe.com> Mon, 12 August 2019 22:09 UTC

Return-Path: <prvs=01273cf045=giuseppe.scaglione@hpe.com>
X-Original-To: tsvwg@ietfa.amsl.com
Delivered-To: tsvwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 33B9A120BD3 for <tsvwg@ietfa.amsl.com>; Mon, 12 Aug 2019 15:09:46 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.6
X-Spam-Level:
X-Spam-Status: No, score=-2.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9E6LuGyTN4qA for <tsvwg@ietfa.amsl.com>; Mon, 12 Aug 2019 15:09:43 -0700 (PDT)
Received: from mx0a-002e3701.pphosted.com (mx0a-002e3701.pphosted.com [148.163.147.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id ED0FE120BD4 for <tsvwg@ietf.org>; Mon, 12 Aug 2019 08:28:40 -0700 (PDT)
Received: from pps.filterd (m0134421.ppops.net [127.0.0.1]) by mx0b-002e3701.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x7CFL9hB004125; Mon, 12 Aug 2019 15:28:35 GMT
Received: from g9t5008.houston.hpe.com (g9t5008.houston.hpe.com [15.241.48.72]) by mx0b-002e3701.pphosted.com with ESMTP id 2ub6wj1t9s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 12 Aug 2019 15:28:34 +0000
Received: from G2W6311.americas.hpqcorp.net (g2w6311.austin.hp.com [16.197.64.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by g9t5008.houston.hpe.com (Postfix) with ESMTPS id 21A7C60; Mon, 12 Aug 2019 15:28:34 +0000 (UTC)
Received: from G9W8671.americas.hpqcorp.net (16.220.49.30) by G2W6311.americas.hpqcorp.net (16.197.64.53) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 12 Aug 2019 15:28:33 +0000
Received: from G4W10204.americas.hpqcorp.net (2002:10cf:5210::10cf:5210) by G9W8671.americas.hpqcorp.net (2002:10dc:311e::10dc:311e) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 12 Aug 2019 15:28:33 +0000
Received: from NAM01-SN1-obe.outbound.protection.outlook.com (15.241.52.10) by G4W10204.americas.hpqcorp.net (16.207.82.16) with Microsoft SMTP Server (TLS) id 15.0.1367.3 via Frontend Transport; Mon, 12 Aug 2019 15:28:33 +0000
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YzY1mlMKKqmZasDUj9KWxMBcQQW6mj9bSwu5GNN5IzE9rjDLFlySj4QmMsgSAn6vhJrPWOi6bmICpAvS1+py1pCnRtbop1Hep/fjk0H3lSLlIq69Xd9+gwiYLXE4p3fhsIivSPRW5kx4ta9fkzS0DsG+w+CWpDSGK3jR1r9l0ys3C6qV5I7SeMKvAFMUCuara3pW3U9VBUmOLccQbr1BFlCwp4x/V5QMfToEOZ2YVqM+Ecqji3ir3LZO7SKDtvV0IOZ17x+N1FeLSSvXVWs7qSAlfmlzWALgsgEEx7axvLCS7KfoV8X7uYkWP+tme50TCQHQuX8tXd6ruMPd+da1RQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=p+R/mAjrW2WsaAwy8cSZCa1MHl/Hl1pRs4eA4utq7WQ=; b=hKFP4A15N5QNLuT41Tnn4IqgA3yySDk0pZi/K5EPnaKQ43sUPlyK7SBLYXGkztjuZL6Wgg576pGvLE2ixDf3kEKc2KLG+y4PmhBoBRKnojn85W7lkvn9zL+2bUhbtVhNKvbhU+IlQxPF6kbZ/Sob+lPEkRNOd5WUt2akmk5j9k82gtVmxm95wkhXE28kRM/MqEzHuxGiC0xpx2S2X0yTGH/O7l+x+O5IVPMImaf819GeOdFh4hqnYoT1oym6zKFLpX+Q1roem7Jq4vhPDqrmwB6dsNt4OgQQdijmuGuUADbLynDogQ65Qq3PkWKBJizup5NlA5KhjKBAOmWhNdrKGg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=hpe.com; dmarc=pass action=none header.from=hpe.com; dkim=pass header.d=hpe.com; arc=none
Received: from AT5PR8401MB0707.NAMPRD84.PROD.OUTLOOK.COM (10.169.8.20) by AT5PR8401MB0338.NAMPRD84.PROD.OUTLOOK.COM (10.169.6.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2157.20; Mon, 12 Aug 2019 15:28:32 +0000
Received: from AT5PR8401MB0707.NAMPRD84.PROD.OUTLOOK.COM ([fe80::d7a:d9a8:652c:bec9]) by AT5PR8401MB0707.NAMPRD84.PROD.OUTLOOK.COM ([fe80::d7a:d9a8:652c:bec9%3]) with mapi id 15.20.2157.022; Mon, 12 Aug 2019 15:28:32 +0000
From: "Scaglione, Giuseppe" <giuseppe.scaglione@hpe.com>
To: Greg White <g.white@CableLabs.com>, Jonathan Morton <chromatix99@gmail.com>
CC: "rgrimes@freebsd.org" <rgrimes@freebsd.org>, "tsvwg@ietf.org" <tsvwg@ietf.org>
Thread-Topic: [tsvwg] Switch testing at 25G with ECN --SCE Draft
Thread-Index: AQHVTjzVagH3r6fcc06dVlu/ZJB8+abx5uIAgAACfxCAAON5gIAAekXQgAAGzgD//6YAAIAAZZ2w//+fYwCAAGWu8A==
Date: Mon, 12 Aug 2019 15:28:32 +0000
Message-ID: <AT5PR8401MB0707095986FAEB5F212F7B5199D30@AT5PR8401MB0707.NAMPRD84.PROD.OUTLOOK.COM>
References: <A8E3F5E9-443D-4F5A-9336-9A0E2E72C278@cablelabs.com> <201908082333.x78NXS0T094756@gndrsh.dnsmgr.net> <AT5PR8401MB07070C672C9F519C05D2D3F599D60@AT5PR8401MB0707.NAMPRD84.PROD.OUTLOOK.COM> <81354E22-777C-4BDB-96E0-0B1F6C1DCCD2@cablelabs.com> <AT5PR8401MB070700703FBF2318808238CF99D60@AT5PR8401MB0707.NAMPRD84.PROD.OUTLOOK.COM> <C9E9A1C7-59CF-4414-82CE-14ABFE74ADB2@gmail.com> <ADCEE0A5-6A32-4106-8557-029C65B2D4C5@cablelabs.com> <AT5PR8401MB07074B665CF7612FAD66E13899D60@AT5PR8401MB0707.NAMPRD84.PROD.OUTLOOK.COM> <6D40C855-EE12-4148-9EE6-E67DE8ADC715@cablelabs.com>
In-Reply-To: <6D40C855-EE12-4148-9EE6-E67DE8ADC715@cablelabs.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [15.211.195.7]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 4e6d4bc6-1359-4378-3f77-08d71f39bc21
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020); SRVR:AT5PR8401MB0338;
x-ms-traffictypediagnostic: AT5PR8401MB0338:
x-microsoft-antispam-prvs: <AT5PR8401MB03387808AD6E7492C4E9BBFD99D30@AT5PR8401MB0338.NAMPRD84.PROD.OUTLOOK.COM>
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-forefront-prvs: 012792EC17
x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(396003)(376002)(346002)(136003)(366004)(39860400002)(189003)(199004)(13464003)(102836004)(8936002)(86362001)(186003)(76176011)(53546011)(6506007)(7696005)(76116006)(66946007)(9686003)(66476007)(66556008)(64756008)(66446008)(446003)(33656002)(55016002)(66066001)(11346002)(486006)(26005)(476003)(229853002)(6436002)(2906002)(478600001)(5660300002)(3846002)(14444005)(6116002)(8676002)(256004)(54906003)(81156014)(81166006)(74316002)(110136005)(99286004)(71190400001)(71200400001)(305945005)(7736002)(52536014)(316002)(14454004)(53936002)(6246003)(25786009)(4326008); DIR:OUT; SFP:1102; SCL:1; SRVR:AT5PR8401MB0338; H:AT5PR8401MB0707.NAMPRD84.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: hpe.com does not designate permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: OInyHljFfK37RdG4T6bBj92yCjicq5SNTRbc1oEyXR36V9fIzm7JUONMkcAzGMK7yC59OUjSGBZzbwbjevmMx8KpLizsKzAHHK2Yl7H0cmaZkfZXo2E/75TCJZWgJ2pYrxiVCrXJ3xktsqxBd/Qy3TkPS+RJC7WDzq0bFDSO/7tapv2Ncu82JwDoG0pCTX7XTtrcW+TOKLbcnjxZova7WtzDPR+S55l+gBTzh3s4Fv0tQdzXLe7fIlfc0AeXc06SjV50mFbUway/g6bkgj3HBNcd+4Sy7Hoe6c8eNG3/G+f8VCoIzxa0YoDfW0Z7oZDZDlYlUVb5im1VrzEB/fDLiKJI3hcs5y2cHTXGVBvxxupIuf5bte0+B1kw1zcHTZ11khVEXPUXrd9FWgJ+F/TS2xLB6bsjuXiOtbqdXNqqsPA=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="utf-8"
X-MS-Exchange-CrossTenant-Network-Message-Id: 4e6d4bc6-1359-4378-3f77-08d71f39bc21
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Aug 2019 15:28:32.4727 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 105b2061-b669-4b31-92ac-24d304d195dc
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 3xGdWwbeFaDcxINBN8Oxio5L7spc7pAaN933EzrC/H1ckltp+5whsOIoPaNZt3gjelG4DWx6U8shprhN42pysfOKYWnXyE8a1GlVS9fIVTs=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AT5PR8401MB0338
X-OriginatorOrg: hpe.com
Content-Transfer-Encoding: base64
X-Proofpoint-UnRewURL: 0 URL was un-rewritten
MIME-Version: 1.0
X-HPE-SCL: -1
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-08-12_06:, , signatures=0
X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1906280000 definitions=main-1908120171
Archived-At: <https://mailarchive.ietf.org/arch/msg/tsvwg/fPFTxZT5NazoPp0EIZdHcQFEt3A>
Subject: Re: [tsvwg] Switch testing at 25G with ECN --SCE Draft
X-BeenThere: tsvwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <tsvwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tsvwg/>
List-Post: <mailto:tsvwg@ietf.org>
List-Help: <mailto:tsvwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 12 Aug 2019 22:09:46 -0000

Hi Greg,

My humble opinion based on the testing presented is that with SCE remarking purely function of the queue depth, the TCP stack has a notion of congestion present in the TCP connection that is independent of the buffering capability of the switch queue. The TCP stack is seeing a "5%" of packet remarked for example, which means somewhere in the network some queue is seeing "some" oversubscription period. 

While with CE RFC3168 you know that remarking typically only starts after a X% of queue depth (user configured). And that X is function of the queue depth and memory available, and if the X is not 'tuned' you either get not optimal link utilization or packet drops. 

Basically, it seems like having an 'early warning' marking makes the SW implementation of the TCP algo easier and more predictable, instead of relying on a switch configuration of X.

Yes -- more testing is required.

Best Regards,
Giuseppe  

-----Original Message-----
From: Greg White [mailto:g.white@CableLabs.com] 
Sent: Friday, August 9, 2019 2:54 PM
To: Scaglione, Giuseppe <giuseppe.scaglione@hpe.com>om>; Jonathan Morton <chromatix99@gmail.com>
Cc: rgrimes@freebsd.org; tsvwg@ietf.org
Subject: Re: [tsvwg] Switch testing at 25G with ECN --SCE Draft

Agreed.  I would not expect any different result either, which begs the question why does SCE need two different signals (ECT(1) and CE) in a datacenter environment?

-Greg



On 8/9/19, 3:43 PM, "Scaglione, Giuseppe" <giuseppe.scaglione@hpe.com> wrote:

    Greg,
    
    We are working -- and in beta -- with having the switch natively set SCE bits instead of CE and removing the iptable configuration on the target server.  Yet, I do not expect any different result since the TCP-SCE stack would react the same.
    
    Regards,
    Giuseppe Scaglione
    
    
    -----Original Message-----
    From: Greg White [mailto:g.white@CableLabs.com] 
    Sent: Friday, August 9, 2019 2:36 PM
    To: Jonathan Morton <chromatix99@gmail.com>om>; Scaglione, Giuseppe <giuseppe.scaglione@hpe.com>
    Cc: rgrimes@freebsd.org; tsvwg@ietf.org
    Subject: Re: [tsvwg] Switch testing at 25G with ECN --SCE Draft
    
    Right.  Per the SCE method, the switch would mark ECT(1) using a ramp, and then when the ramp would exceed 100% marking, it would change to using CE.  
    
    The implementation discussed here marks CE using a ramp, and when the ramp would exceed 100% marking, it changes to packet drop.  This, as you said, is simply RFC3168 ECN marking, not SCE ECN marking.
    
    I was just trying to set the record straight.  There was a claim made that a switch vendor had implemented SCE-style packet marking in hardware at 25Gbps, which wasn't accurate.
    
    -Greg
    
    
    
    On 8/9/19, 2:58 PM, "Jonathan Morton" <chromatix99@gmail.com> wrote:
    
        > On 9 Aug, 2019, at 11:44 pm, Scaglione, Giuseppe <giuseppe.scaglione@hpe.com> wrote:
        > 
        >>> Just to be super clear, this isn't a hardware implementation of SCE running at 25Gbps.
        > 
        > I am not sure I follow. The Test setup section of the paper clearly describes the hardware used -- severs, switch, cables. 
        > What I cannot disclose at this point is the exact model and characteristics of the HPE Aruba Switch used. Yet, it is a "real" Ethernet Switch, providing 25Gbps connectivity, configured to do bridging across the four ports and implementing RFC3168 with the ECN remarking configuration described in my previous email and on the paper.
        
        I think the issue is with the way the switch itself marks with CE, not ECT(1).  It's a limitation I think is worth acknowledging and, hopefully, finding a way to remove.
        
         - Jonathan Morton