Re: [Ntp] Hard NO: Re: WGLC - draft-ietf-ntp-ntpv5-requirements

[David Venhoek <[e-mail]>]

Apologies for the delay, processing the chrony data took me a lot
longer due to a variety of factors, but anyway here the same
statistics for chrony:

standard deviation: 2.4us
90 percent is within: 3.8 us
95 percent is within: 4.6 us
99 percent is within: 7.6 us

The same caveats mentioned in the previous email apply.

Kind regards,
David Venhoek

On Fri, Jan 12, 2024 at 10:41 AM David Venhoek <[e-mail]> wrote:
>
> Apologies for this taking a while, but between needing access to the
> office (which was closed over the holidays) to get clean data
> demonstrating my point (I have internal data that shows the same, but
> it isn't to a standard I would expect others to accept), and making
> some mistakes it took me a while to get measurement data to share.
>
> The data presented here is for ntpd-rs and the reference
> implementation, chrony is still running as we speak and I will send it
> sometime next week when I have time. My argumentation for the order of
> magnitude improvement is this data and the comparison published at
> https://chrony-project.org/comparison.html.
>
> What I tested was the following specific situation: A endrun ninja gps
> reference clock running ntp, providing time to a raspberry pi 3 via a
> HP aruba switch. Each of ntpd-rs and the reference implementation was
> configured to use the endrun ninja as it's only remote time reference,
> with poll interval of 8 seconds. The raspberry pi 3 and the endrun
> ninja were both configured to output a pulse per second signal, the
> offset between these was measured.
>
> For each test, the implementation under test was started and given 2
> hours to stabilize, and then the offsets between rpi3 and ninja was
> measured for 4 hours.
>
> Below are provided for each the standard deviation from mean of the
> offset, and 90, 95 and 99 percent worst offsets from the mean offset.
> Mean offset was ignored because there are multiple known sources of
> assymetries in the system, including from the ninja reference clock
> and the switch, and I haven't done sufficient analysis of all sources
> to be able to attribute a quantity to the ntp implementation
> specifically (hence making this value rather meaningless). Furthermore
> assymetries in general are not a function of any particular algorithm,
> but rather of how the timestamping is done, which best I can tell is
> not standardized in teh spec and hence not part of the current
> discussion. Please take the p99 with a grain of salt, as with the 4
> hour measuring period the amount of data supporting that is somewhat
> on the thin side.
>
> Attached as images are plots of the allan deviation and offsets of the
> clock when controlled by each of the implementations in the range
> 1-2048 seconds.
>
> NTP reference implementation
> -----------------------------
>
> standard deviation: 36.5 us
> 90 percent is within: 24.6 us
> 95 percent is within: 59.4 us
> 99 percent is within: 192.2 us
>
> ntpd-rs
> -------
>
> standard deviation: 3.4 us
> 90 percent is within: 5.5 us
> 95 percent is within: 6.4 us
> 99 percent is within: 8.9 us
>
>
> Hope that answers your question regarding what I base my assertions
> on. If you are interested in further measurements please let me know,
> then we can discuss what we can do practically. Attached below are
> some of the technical bounds on errors in the measurement itself for
> those interested.
>
> For those wanting to play around with the data themselves, the
> measurement offset data is also attached. The format is one line per
> second, in order, with the offset in nanoseconds.
>
> Kind regards,
> David Venhoek
>
>
>
> Uncertainty estimates on the measurements:
>
> In the setup, we have identified the following sources of error:
> Cable length differences: We have differences in cable length between
> various parts of at most 1 meter. At 0.3c signal propagation speed
> this represents a 12 ns uncertainty
> Time interval measurement: The time difference measurement aparatus
> has a resolution of 1 ns, validated to 10 ns. Furthermore the clocking
> used internally gives a 0.01% relative uncertainty. For the range
> investigated here this represents an uncertainty of 30 ns
> endrun ninja pps edge uncertainty: Has been validated to be within
> 100ns of gps input.
> Rpi3 pps edge uncertainty: Has been validated to be within 100ns of
> internal clock.
>
> In combination, the measurements are accurate to at least 300 ns.
> Confidence bounds on the derived statistics depend significantly on
> assumptions on the underlying distribution, but the differences are
> significant enough to be unlikely to be chance for the standard
> deviation, p90 and p95 statistics. For the p99 statistic, the sample
> size is likely on the small side, especially given correlation between
> the individual measurements.
>
> On Thu, Dec 28, 2023 at 7:22 AM Dave Hart <[e-mail]> wrote:
> >
> > On Wed, 27 Dec 2023 at 13:04, David Venhoek <[e-mail]> wrote:
> >>
> >> >From my perspective, the fact that apparently (at least as harlan
> >> seems to claim in his original email) the algorithm specified in
> >> RFC5905 is fragile would be all the more reason to not specify it as
> >> "the standard". If this is really the case, we should rather be
> >> looking harder for solutions that are more resilient and not so
> >> sensitive to
> >
> >
> > Fortunately this is not the case.  Harlan did not claim in his original email of this thread that NTPv4 is fragile.  Your apparent misunderstanding may come from his mention that Dr. Mills showed that _violating_ the specification can easily break a synchronization.  That is quite the opposite of claiming RFC5905's algorithms are fragile.
> >
> >> [...]
> >>
> >> Looking at the current moment, we are now in a situation where there
> >> are 3 algorithms operating in the space, with the one in the
> >> "reference" implementation, the one from chrony and the one from
> >> ntpd-rs. I have personally not seen any indication of problems caused
> >> by this, rather the oposite. From measurements it looks very likely
> >> that both chrony and ntpd-rs are capable of, with similar poll
> >> intervals, synchronizing to about an order of magnitude larger
> >> precision.
> >
> >
> > Your use of precision is confusing in this context.  In NTP parlance, precision is the minimum change in the system clock, in ancient systems, the clock tick, in modern systems, the time to read the system clock.  I look forward to you documenting your claim of an order of magnitude better performance from Chrony and ntpd-rs compared to the reference implementation.  I would love to improve the performance of NTP across a wide variety of situations and then promote standardizing those improvements.
> >>
> >> [...]
> >>
> >> Also, the fact that David L Mills apparantly had the tendency to just
> >> change the algorithm without changes to the standard at least on the
> >> surface feels also just wrong to me. That gives a lot of vibes of
> >> "innovation, but only by us" from the "reference" implementation
> >> people, which is I think highly harmful. I sincerely hope that that is
> >> just a wrong impression from my side.
> >
> >
> > It sounds like you're not familiar with the history of NTP.  In the beginning, there were fuzzball routers and Arpanet...  I don't think it would be productive for me to spend a lot of time repeating what can be easily discovered with a little effort.  I refer you to Dr. Mills' papers at https://www.eecis.udel.edu/~mills/, the NTP RFCs, and his second edition NTP book "Computer Network Time Synchronization: The Network Time Protocol on Earth and in Space, Second Edition 2nd Edition" ISBN 978-1439814635.
> >
> > Suffice it to say Mills invented NTP and nurtured it personally for 30 years.  He maintained various iterations of the reference implementations over that timeframe, eventually with the assistance of Harlan Stenn.  Dr. Mills' health no longer allows him to participate substantially.  During that time, he naturally evolved the design _before_ codifying it in standards, and I know of no other reasonable approach.  He never wavered from the position that NTP was not simply a wire protocol, but a suite of algorithms carefully engineered, simulated, and tested over time to have predictable and consistently beneficial behavior in synchronizing networks of devices within the limits of their oscillators, reference clocks, communications channels, hardware, and underlying software.
> >
> > If you want to use the on-wire protocol without the algorithms, please see the SNTP RFCs.
> >
> > Cheers,
> > Dave Hart
> >
> >
> >>
> >> On Wed, Dec 27, 2023 at 8:22 AM Harlan Stenn <[e-mail]> wrote:
> >> >
> >> > On 12/26/2023 3:23 PM, Dave Hart wrote:
> >> > > On Thu, 14 Dec 2023 at 17:18, Miroslav Lichvar <[e-mail]
> >> > > <mailto:[e-mail]>> wrote:
> >> > >
> >> > >     On Thu, Dec 14, 2023 at 03:16:29AM -0800, Harlan Stenn wrote:
> >> > >      > The core "mission" of NTP is time synchronization with a (well)
> >> > >     defined
> >> > >      > response to a "time impulse".  This is the reason why previous NTP
> >> > >      > specifications have included the algorithms.  Prof. Mills and
> >> > >     some others
> >> > >      > have done a LOT of testing to ensure reliable and predictable
> >> > >     behavior of
> >> > >      > time synchronization, in the "normal" and "time impulse" cases
> >> > >     over a very
> >> > >      > wide range of circumstances.
> >> > >
> >> > >     If the RFC 5905 PLL+FLL is so great, why is nothing using it, not even
> >> > >     the "reference" implementation in default configuration?
> >> > >
> >> > >
> >> > > Would you mind elaborating how the reference implementation's PLL+FLL
> >> > > feedback loop differs from the NTPv4 spec?  I'm not aware of any
> >> > > intentional deviation, but Dr. Mills wasn't shy about making changes to
> >> > > the implementation that he felt was an improvement before documenting it
> >> > > in another RFC.
> >> > >
> >> > >     ntpd in default configuration has a poor response with longer polling
> >> > >     intervals. It suffers from oscillations,
> >> > >
> >> > >
> >> > > If verified that would seem to me a reason to improve the algorithms,
> >> > > rather than decide it's time for a wild west where every NTP
> >> > > implementation is free to behave in any way, as that would invite
> >> > > pathological results in situations where differing implementations sit
> >> > > on the synchronization path between the reference clock and the ultimate
> >> > > client.
> >> >
> >> > Or better describe the conditions for these problems?
> >> >
> >> > If you are saying that the default config can show oscillations as poll
> >> > intervals increase, all I can say is we haven't seen reports of this.
> >> >
> >> > If we had, we'd be taking steps to fix it.
> >> >
> >> > If you have seen this, perhaps you'd be kind enough to post about how
> >> > one might change the default values to ones more suitable for longer
> >> > poll intervals, or even telling us how to demonstrate the problem.
> >> >
> >> > >     which can be sometimes seen
> >> > >     even on monitoring graphs of pool.ntp.org <http://pool.ntp.org>.
> >> > >
> >> > >
> >> > > That public pool uses primitive monitoring that does not take into
> >> > > account the delay or jitter between the monitoring station and the
> >> > > server.  Moreover, the requirements for participating are very lenient,
> >> > > allowing clocks that appear to be up to 70ms off of UTC.  That pool is
> >> > > therefore not a good example of a well-engineered and well-maintained
> >> > > synchronization source.  It's fine to get the clock within a few hundred
> >> > > milliseconds, but stricter requirements call for a more precise source
> >> > > and better error budgeting.
> >> > >
> >> > >     Nobody seems to care. Maybe
> >> > >     it's a bug, but after so many years I think we can conclude that
> >> > >     Internet will not break if all NTP implementations don't have the
> >> > >     "well defined" response.
> >> > >
> >> > >
> >> > > The internet will not break even if all NTP sources were only good to a
> >> > > few seconds.  Those who require tight sync (such as distributed
> >> > > databases) engineer solutions to meet their requirements.
> >> >
> >> > I'm negatively impressed with your conclusion, Miroslav.
> >> >
> >> > "The Internet" probably won't break, because "the internet" doesn't
> >> > exchange time that way, and I would bet that you know this.
> >> >
> >> > A single machine will either have a crafted config file, well-tended or
> >> > not, and static or pool servers.  How well do you think the vast
> >> > majority of these machines are monitored to see if there are problems?
> >> > How badly would they have to screw up to be noticed?
> >> >
> >> > If somebody bothers to look and sees one of the hosts in their static
> >> > config file is bad, they will likely just throw out the bad site and
> >> > replace it.
> >> >
> >> > If they are using the "pool" directive and there are misbehaving servers
> >> > *that otherwise survive the pool monitoring service* then ntpd will
> >> > notice the bad performers and throw them out automatically.
> >> >
> >> > In an enterprise, the odds are quite high that time for the enterprise
> >> > is sync'd from a set of curated machines.  These machines are likely
> >> > getting their time from reliable sources.  They won't be talking to
> >> > poorly-behaving time sources.  This translates to the (internal)
> >> > machines that get their time from the (well-behaved/reliable) internal
> >> > time sources.
> >> >
> >> > So sure, the stuff the NTP Project has put out there is very resilient
> >> > and well-behaved.  There's a good chance it will continue to behave well
> >> > even in an increasingly hostile environment.
> >> >
> >> > But why would any <positive-intentioned> person want to take steps to
> >> > increase the environment's hostility?
> >> >
> >> > As I have said before, the world of time-synchronization is not the
> >> > place to use creative destruction as a method to promote evolution.
> >> >
> >> > > Cheers,
> >> > > Dave Hart
> >> > >
> >> > >
> >> > > _______________________________________________
> >> > > ntp mailing list
> >> > > [e-mail]
> >> > > https://www.ietf.org/mailman/listinfo/ntp
> >> >
> >> > --
> >> > Harlan Stenn <[e-mail]>
> >> > http://networktimefoundation.org - be a member!
> >> >
> >> > _______________________________________________
> >> > ntp mailing list
> >> > [e-mail]
> >> > https://www.ietf.org/mailman/listinfo/ntp
> >>
> >> _______________________________________________
> >> ntp mailing list
> >> [e-mail]
> >> https://www.ietf.org/mailman/listinfo/ntp
> >
> >
> >
> > --
> > Cheers,
> > Dave Hart