Re: The TCP and UDP checksum algorithm may soon need updating
Craig Partridge <craig@tereschau.net> Sat, 06 June 2020 23:17 UTC
Return-Path: <craig@tereschau.net>
X-Original-To: ietf@ietfa.amsl.com
Delivered-To: ietf@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7A85E3A02C1 for <ietf@ietfa.amsl.com>; Sat, 6 Jun 2020 16:17:29 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.088
X-Spam-Level:
X-Spam-Status: No, score=-2.088 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_FILL_THIS_FORM_SHORT=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=tereschau.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RFioS828D-H9 for <ietf@ietfa.amsl.com>; Sat, 6 Jun 2020 16:17:27 -0700 (PDT)
Received: from mail-qk1-x72c.google.com (mail-qk1-x72c.google.com [IPv6:2607:f8b0:4864:20::72c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 23B593A00D2 for <ietf@ietf.org>; Sat, 6 Jun 2020 16:17:27 -0700 (PDT)
Received: by mail-qk1-x72c.google.com with SMTP id n141so13734872qke.2 for <ietf@ietf.org>; Sat, 06 Jun 2020 16:17:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tereschau.net; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=uY5pZ1M02PgEp9MsnmosmqHQLovOLmFPOrPURXWqx/c=; b=kNi1Uyrkuu/lhmifvYPU1O9ZVN2VunLnyCSh8EcLDas1gEav2bcchBO25OizCVAizp 4fRt/EE8zD+GWJCxZAGpYSFm0Gjui8aHw6hxrzgEHcGP6PrNKN5B6u5bsApU9g9ycFeO ZOXcqZK4TnBVJdQGGIHfmJiJ8LQu/a4IKnE5636JxOPRHI5bpQ/OLJC5qGmUl3U3qQQ8 qAPmv4csMoZ0WMqA5XlfEHQQDv+bBVt5eUUNN2v6IMQ3C9kP7ZQOdDfZBj178GFpOMB/ COa4Hrr26FyVlGBPp/7GPVGT9SLJMx2n2OrU0LGutgjvEceIGRgKezRi3Q5UtfHd5+tJ hQRw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=uY5pZ1M02PgEp9MsnmosmqHQLovOLmFPOrPURXWqx/c=; b=aqN1DkPAHj0trTRZSLRNw8PEVUlAwhZHm3CDgLhCjk2MlLHyb+FWoUtijm1MStrTOn KlluMFOCppOB+yN4JDZtU+KZ2NrWXNNiSDWE82l536/j7y8CFIZSlb4KyAFmJcj+hiGx 4/VHoQuuoID4Kr1arV4IiSuw+7YGeCWJAmPsa+TQUYveg3Lz5zQ8jQKclGHxOx9qQWmT gSWV1NGKqpzI32Dtx9NEHu71cy1URouiwya4bP9siF9gksl1XM3iVgPuWbpz5RfLqe4U eOp/5ATKcBXxAH7q6UENz7MGiQf61ipwRc+dIxXZkAVIxdXbaJnhLrvnVuy73zWX0XGf NfCw==
X-Gm-Message-State: AOAM531f0tGgxdS43cFZEOEyU/AIsq5Uho4EsxZuwebK++SHgSF7TMNd 5ZmqjBWBVVb9P8l8VGaZnha7vQreWclE9RxnzxGHDw/wf2w=
X-Google-Smtp-Source: ABdhPJyA8uAtmSf8UZOiuT7sp3LUrYxIxJG/QS+dgIBwpdIznV5q1T4AwHYceQA5XrqvxCRX8H+g4s9VXxQcc8+30js=
X-Received: by 2002:a37:2c85:: with SMTP id s127mr13337862qkh.35.1591485444183; Sat, 06 Jun 2020 16:17:24 -0700 (PDT)
MIME-Version: 1.0
References: <CAHQj4Cf_vgXYEL=x4DCEnpwNxZpJQSD-h6MWmhMWpYwPF9XFow@mail.gmail.com> <0D18B54B-2865-4A3C-813B-595EA17F6D8B@gmail.com> <32750.1591376396@localhost>
In-Reply-To: <32750.1591376396@localhost>
From: Craig Partridge <craig@tereschau.net>
Date: Sat, 06 Jun 2020 17:17:13 -0600
Message-ID: <CAHQj4CdopwpEfyuOVO3ZywTKveQMpnt_WPh_JDRydgNKHVVmhw@mail.gmail.com>
Subject: Re: The TCP and UDP checksum algorithm may soon need updating
To: IETF discussion list <ietf@ietf.org>
Content-Type: multipart/alternative; boundary="00000000000090ef8a05a772942e"
Archived-At: <https://mailarchive.ietf.org/arch/msg/ietf/53ROo-7Ix_ax8DoPGVvkJxZDIw0>
X-BeenThere: ietf@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF-Discussion <ietf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf>, <mailto:ietf-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ietf/>
List-Post: <mailto:ietf@ietf.org>
List-Help: <mailto:ietf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf>, <mailto:ietf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 06 Jun 2020 23:17:30 -0000
I got requests for citations/sources. So here's what's known. First, background reading from 20 years ago -- shows the distribution of errors that we saw in TCP connections then. Note while bit errors happen, far more are bytes or larger. This paper was also the origin of the idea of putting 32-bit md5 sums on files... @inproceedings{10.1145/347059.347561, author = {Stone, Jonathan and Partridge, Craig}, title = {When the CRC and TCP Checksum Disagree}, year = {2000}, isbn = {1581132239}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = { https://doi.org/10.1145/347059.347561}, doi = {10.1145/347059.347561}, booktitle = {Proceedings of the Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication}, pages = {309–319}, numpages = {11}, location = {Stockholm, Sweden}, series = {SIGCOMM ’00} } OK, on to what people are seeing today. This shows that 1 in every 121 file transfers FTP delivers a file that, when you do the md5 sum, turns out not to match the original (note there are multiple possible reasons, but TCP checksum is a strong candidate). @inproceedings{Liu:2018:CSD:3208040.3208053, author = {Liu, Zhengchun and Kettimuthu, Rajkumar and Foster, Ian and Rao, Nageswara S. V.}, title = {Cross-geography Scientific Data Transferring Trends and Behavior}, booktitle = {Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing}, series = {HPDC '18}, year = {2018}, isbn = {978-1-4503-5785-2}, location = {Tempe, Arizona}, pages = {267--278}, numpages = {12}, url = {http://doi.acm.org/10.1145/3208040.3208053}, doi = {10.1145/3208040.3208053}, acmid = {3208053}, publisher = {ACM}, address = {New York, NY, USA}, keywords = {GridFTP, file transfer, usage management, wide area network}, } On a related point, about 60% of big file transfers in a major energy network are failing (checksums are one of the suspects); @inproceedings{shannigrahi2017request, title={Request aggregation, caching, and forwarding strategies for improving large climate data distribution with NDN: a case study}, author={Shannigrahi, Susmit and Fan, Chengyu and Papadopoulos, Christos}, booktitle={Proceedings of the 4th ACM Conference on Information-Centric Networking}, pages={54--65}, year={2017}, organization={ACM} } This reference also sees high error rates: Kettimuthu, Rajkumar, et al. "Transferring a Petabyte in a Day." Future Generation Computer Systems 88 (2018): 191-198. Anecdotally, folks are reporting some middlebox vendors are not updating the TCP checksum but rather letting the outbound interface simply recompute the entire checksum -- which means that if the TCP segment gets damaged during middlebox handling, the middlebox will slap a valid checksum on bad data. As I noted -- the literature is slim, which is why a team I'm on is going to seek more comprehensive error collection which actually captures the errors in the data, so we can see what kinds of errors are causing trouble. Craig -- ***** Craig Partridge's email account for professional society activities and mailing lists.
- The TCP and UDP checksum algorithm may soon need … Craig Partridge
- Re: The TCP and UDP checksum algorithm may soon n… Joe Touch
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… Craig Partridge
- Re: The TCP and UDP checksum algorithm may soon n… Richard Barnes
- Re: The TCP and UDP checksum algorithm may soon n… Craig Partridge
- Re: The TCP and UDP checksum algorithm may soon n… Brian E Carpenter
- Re: The TCP and UDP checksum algorithm may soon n… Nico Williams
- Re: The TCP and UDP checksum algorithm may soon n… Nico Williams
- Re: The TCP and UDP checksum algorithm may soon n… Richard Barnes
- Re: The TCP and UDP checksum algorithm may soon n… Richard Barnes
- Re: The TCP and UDP checksum algorithm may soon n… Christian Huitema
- Re: The TCP and UDP checksum algorithm may soon n… Christian Huitema
- Re: The TCP and UDP checksum algorithm may soon n… Phillip Hallam-Baker
- Re: The TCP and UDP checksum algorithm may soon n… Joseph Touch
- Re: The TCP and UDP checksum algorithm may soon n… Stewart Bryant
- Re: The TCP and UDP checksum algorithm may soon n… Phillip Hallam-Baker
- Re: The TCP and UDP checksum algorithm may soon n… Joseph Touch
- Re: The TCP and UDP checksum algorithm may soon n… Nico Williams
- Re: The TCP and UDP checksum algorithm may soon n… Michael Richardson
- Re: The TCP and UDP checksum algorithm may soon n… Nico Williams
- Re: The TCP and UDP checksum algorithm may soon n… Craig Partridge
- Re: The TCP and UDP checksum algorithm may soon n… Masataka Ohta
- Re: The TCP and UDP checksum algorithm may soon n… John Levine
- Re: The TCP and UDP checksum algorithm may soon n… Phillip Hallam-Baker
- Re: The TCP and UDP checksum algorithm may soon n… Joseph Touch
- Re: The TCP and UDP checksum algorithm may soon n… Christian Huitema
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… Joseph Touch
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… Salz, Rich
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… Salz, Rich
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… Michael Richardson
- Re: The TCP and UDP checksum algorithm may soon n… Benjamin Kaduk
- Re: The TCP and UDP checksum algorithm may soon n… Nick Hilliard
- Re: The TCP and UDP checksum algorithm may soon n… Carsten Bormann
- Re: The TCP and UDP checksum algorithm may soon n… Nick Hilliard
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… Joe Touch
- Re: The TCP and UDP checksum algorithm may soon n… Nico Williams
- Re: The TCP and UDP checksum algorithm may soon n… Nico Williams
- Re: The TCP and UDP checksum algorithm may soon n… Nick Hilliard
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… Nick Hilliard
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… Carsten Bormann
- Re: The TCP and UDP checksum algorithm may soon n… Nico Williams
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… Nico Williams
- Re: The TCP and UDP checksum algorithm may soon n… John C Klensin
- Re: The TCP and UDP checksum algorithm may soon n… Nico Williams
- Re: The TCP and UDP checksum algorithm may soon n… Richard Barnes
- Re: The TCP and UDP checksum algorithm may soon n… Russ Housley
- Re: The TCP and UDP checksum algorithm may soon n… Brian E Carpenter
- Re: The TCP and UDP checksum algorithm may soon n… Nico Williams
- Re: The TCP and UDP checksum algorithm may soon n… Warren Kumari
- Re: The TCP and UDP checksum algorithm may soon n… John C Klensin
- Re: The TCP and UDP checksum algorithm may soon n… Christian Huitema
- Re: The TCP and UDP checksum algorithm may soon n… John C Klensin
- Re: The TCP and UDP checksum algorithm may soon n… Craig Partridge
- Re: The TCP and UDP checksum algorithm may soon n… Nico Williams
- Re: The TCP and UDP checksum algorithm may soon n… Eric Rescorla
- Re: The TCP and UDP checksum algorithm may soon n… Craig Partridge
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… Nico Williams
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… John Levine
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas
- Re: The TCP and UDP checksum algorithm may soon n… Brian E Carpenter
- Re: The TCP and UDP checksum algorithm may soon n… Warren Kumari
- Re: The TCP and UDP checksum algorithm may soon n… John R Levine
- Re: The TCP and UDP checksum algorithm may soon n… tom petch
- Re: The TCP and UDP checksum algorithm may soon n… Salz, Rich
- Re: The TCP and UDP checksum algorithm may soon n… Carsten Bormann
- Re: The TCP and UDP checksum algorithm may soon n… Salz, Rich
- Re: The TCP and UDP checksum algorithm may soon n… Craig Partridge
- Re: The TCP and UDP checksum algorithm may soon n… Michael Thomas