[babel] Accuracy of timestamps for babel-rtt: userspace vs kernel timestamps

[Baptiste Jonglez <[e-mail]>]

Hello,

A recent discussion with Dave convinced me to start looking at whether
very short RTTs make any sense in Babel, and whether they could be used to
infer link speed.  If only to settle theses questions for good.  Since the
related subject of nanosecond-resolution timestamps was brought up by Toke
at the IETF session yesterday, I made a quick test with kernel timestamps
today.

I'll talk about the implementation and shortcomings of using kernel
timestamps below, but here are some rough timing results.  I just used a
veth pair on my laptop (4.17 kernel) with a babeld on each side of the
pair, and didn't do any serious statistics.

- regular babeld: average measured RTT ~320 µs (quite variable)

- babeld with kernel RX timestamps: average measured RTT ~120 µs (quite variable)

- ping through the same veth pair (link-local IPv6, 1000 packets): average 105 µs, minimum 16 µs

The observant reader will notice that the current resolution (1 µs) is
more than enough in that case.  Also, using kernel timestamps improves
accuracy by about 100 µs on each host, a somewhat significant improvement.


Now, regarding the implementation of kernel timestamps in babeld, my test code is here:

  https://github.com/jonglezb/babeld/commit/56756a8cbe9a0b8a168c78873dd77e48e5770278

Thank you Dave for your first draft of this code, I borrowed a bit from it ;)

As explained in the commit message, it suffers from a number of issues
that would need some serious work before it's really usable:

- kernel timestamps use the realtime clock, and it's not configurable.
  This is really annoying because babeld uses the monotonic clock (for
  good reasons).  Using kernel timestamps forces us to fall back to the
  realtime clock elsewhere in babeld, and it would require some work to do
  it cleanly.

- kernel timestamps are only used for received packets.  The sending side
  (timestamp in Hello) still uses userspace timestamps, with the ensuing
  accuracy issue.  It does not seem possible to tell the kernel to embed a
  timestamp at a specified location in a packet just before sending it,
  unless maybe playing with eBPF.


Takeaway: at least on Linux, I don't see a use-case for nanosecond
resolution timestamps.  If somebody ever writes an implementation of Babel
for specialized hardware and runs a datacenter with ultra-low-latency
network equipments, it could possibly still make sense.


-- 
Baptiste Jonglez
PhD student
Univ. Grenoble Alpes <https://www.univ-grenoble-alpes.fr/>
LIG lab <https://www.liglab.fr/>
Drakkar team <http://drakkar.imag.fr/>  |  Polaris team at INRIA <https://team.inria.fr/polaris/>