dataplane encapsulation considerations - checksums

[Stewart Bryant <[e-mail]>]

Changing the thread title to match the subject

Lloyd

Those numbers are orders of magnitude larger than anything I have
seen elsewhere.

If we look at them in detail they are interesting because if the no ports
are c/s errors as you hypothesis, then only 1/3 of the errors are seemingly
caught by the c/s. I would thus expect  you to be arguing strongly
that we need to deprecate the existing c/s in favour of something
much stronger such as fletcher, or ideally a crypto checksum.
In the case of the link state IGPs we have operators that turn on
link state security not for security but for the enhanced checksum
and that seems to be what you need here in the host stacks.

However we have another study that is worth looking at:

https://www.verisigninc.com/assets/VRSN_Bitsquatting_TR_20120320.pdf

This looks at the UDP c/s errors received by DNS servers, and they
saw an error rate of 1 in 10^5. However if you read the detail there are 
some
systematic effects going on, with a significant fraction seemingly due
to transmit host stack problems which may be what you are seeing.

Now I think all this points to two things, firstly from a perspective
of a host the c/s looks inadequate and there may need to be a
revision of the host stack.

On the other hand the reported error rate for UDP that the paper
reports is an over estimate of the error rate in the tunnel case since
only an error in the IP and UDP header can cause misdelivery
with all other errors reflecting themselves as payload errors which
the payload error protection. Although I do note that they report
some systematic effect in terms of bit position which may point either
way.

One other aspect of this, which is important in Routing, if we are seeing
the error rates you imply are getting past the TCP c/s, surely the routing
protocols will be importing long term errors into the routing subsystems
(specifically BGP and LDP). Are people observing that and if so don't we
need to fix it?

- Stewart



On 10/12/2014 13:24, [e-mail] wrote:
> Stewart,
>
> Who needs an NMS? Let's go old school.
> TCP and UDP have equivalent pseudo-header+payload ones complement checksums. So data from TCP can be considered as somewhat equivalent as the technology is the same, and TCP is somewhat better instrumented and monitored than UDP is.
>
> Below sample from two core switches, on tcp connections to them  (not, alas, through them) shows e.g. 10287 TCP checksum errors for 4.3 million TCP/IP packets received. That would be an error rate of 0.24% - the other switch is 0.45% . Note the figure of e.g. 19458 no port - entirely possible something inside the core network is trying a port that isn't there, but also possible corruption of port nos. These are firewalled core devices with limited traffic to them, so I would consider this a best case, without throwing edge device data into the mix.
>
> Extrapolating from TCP checksum rates to UDP checksum rates, from traffic to to traffic through, and from v4 to v6 is left as an exercise for the reader.
>
> Incidentally, these rates pretty much match the 1 in 400 observation made early in Stone's SIGCOMM 2000 paper. 1:400 is 10,000:4,000,000
> http://conferences.sigcomm.org/sigcomm/2000/conf/paper/sigcomm2000-9-1.pdf
> Zut alors, quelle surprise, physics is still physics, this particular SIGCOMM paper is still valid.
> Checksums fail because of corruption. But at least there's a checksum to catch the corruption. Sans checksums, the corruption is unnoticed.
>
> The main reason noone has seen misbehaviour with MPLS is because no-one is looking for or instrumenting for it. (Just like MPLS doesn't need TTL, because no-one ever sees MPLS routing loops, right?) Please go measure MPLS and report back.
>
> Traversing over UDP, as suggested in draft-ietf-mpls-in-udp and draft-ietf-tsvwg-gre-in-udp-encap, increases the scope for corruption across a longer path, rather than individual links - and the UDP checksum is an important last check.
>
> SWT1#sh tcp stat
> Rcvd: 4332545 Total, 19458 no port
>        10287 checksum error, 4959 bad offset, 0 too short
>        62946 packets (2289083 bytes) in sequence
>        93 dup packets (19167 bytes)
>        34 partially dup packets (1878 bytes)
>        107 out-of-order packets (17980 bytes)
>        8 packets (8 bytes) with data after window
>        0 packets after close
>        0 window probe packets, 614 window update packets
>        682 dup ack packets, 0 ack packets with unsend data
>        179823 ack packets (37459618 bytes)
> Sent: 2049509 Total, 0 urgent packets
>        1774234 control packets (including 345 retransmitted)
>        249103 data packets (37439059 bytes)
>        308 data packets (41767 bytes) retransmitted
>        46 data packets (4876 bytes) fastretransmitted
>        25718 ack only packets (3764 delayed)
>        0 window probe packets, 104 window update packets
> 9649 Connections initiated, 1230 connections accepted, 10857 connections established
> 1760675 Connections closed (including 30 dropped, 1749812 embryonic dropped)
> 653 Total rxmt timeout, 9 connections dropped in rxmt timeout
> 0 Keepalive timeout, 6 keepalive probe, 0 Connections dropped in keepalive
>
>
> SWT2#sh tcp stat
> Rcvd: 6370948 Total, 37608 no port
>        28419 checksum error, 1749 bad offset, 0 too short
>        181714 packets (3507560 bytes) in sequence
>        130 dup packets (4925 bytes)
>        164 partially dup packets (302 bytes)
>        35 out-of-order packets (278 bytes)
>        64 packets (4549 bytes) with data after window
>        0 packets after close
>        0 window probe packets, 2006 window update packets
>        3865 dup ack packets, 0 ack packets with unsend data
>        857922 ack packets (92503306 bytes)
> Sent: 3793587 Total, 0 urgent packets
>        2667720 control packets (including 215 retransmitted)
>        1042867 data packets (92804839 bytes)
>        1772 data packets (74842 bytes) retransmitted
>        192 data packets (7176 bytes) fastretransmitted
>        81025 ack only packets (15840 delayed)
>        0 window probe packets, 19 window update packets
> 28735 Connections initiated, 5928 connections accepted, 34638 connections established
> 2626485 Connections closed (including 1349 dropped, 2591827 embryonic dropped)
> 1987 Total rxmt timeout, 4 connections dropped in rxmt timeout
> 0 Keepalive timeout, 0 keepalive probe, 0 Connections dropped in keepalive
>
> Lloyd Wood
> http://about.me/lloydwood
>
> error-free modern networking technology? ha.