Re: [Ntp] CLOCK_TAI (was NTPv5: big picture)

[Magnus Danielson <[e-mail]>]

Martin,

On 2021-01-08 15:52, Martin Burnicki wrote:
> Magnus,
>
> Magnus Danielson wrote:
>> On 2021-01-07 17:20, Martin Burnicki wrote:
>>> My opinion is that if you simplify here, you just shift the problem
>>> elsewhere, and you probably get even more problems if you have to care
>>> about that.
>> But that is the whole point of the exercise! For sure the issue do not
>> disappear, but the hope is that it can move into a corner where it makes
>> less harm.
> If you are not involved in the corner where it is shifted then it's fine
> for you, but not for those involved in the corner.
I had no intention of not caring for that corner. Quite the opposite. I
want the overall system architecture, all the encodings be very explicit
and complete such that all those parts will be clear so that it can be
correctly implemented. It's exactly because of that for which I think
that if we make sure the core timing is easy, without extra checks to
workaround, the resulting output mapping needed can be worth the price,
yet easy and simple enough that even simple implementations can do it
right.
>
> For example, look at the specs of the IRIG codes. I'm sure you know that
> the most commonly used IRIG codes (e.g. B122) only provided a
> day-of-year number and the time of day, but the original IRIG specs
> didn't even provide a way for a receiver to find out whether the
> transported time was something like UTC, or some local time with an
> offset to UTC.
>
> This in why in 1995 the IEEE 1344 standard was released, which used some
> "reserved" bits of the B122 format to provide the time code receivers
> with some useful information, e.g. a 2 digit year number, a local time
> offset that could be used to derive UTC from the transported local time,
> and even some flags like a DST indicator and a leap second warning.
>
> This extension was (and still is) very useful if you do synchronization
> using these time codes. However, 10 year later, when IEEE 1344 was well
> established and known, the specs were revised, and some folks meant that
> the way the time offset is to be applied to convert local time to UTC
> had to be reverted because in other (unrelated) scenarios this would
> also have to be subtracted instead of being added (or vice versa, but
> that doesn't matter here).
>
> Anyway, the revised standard was called C37.118 according to the new
> IEEE naming scheme, the information transported in the code were the
> same as before, plus some more details, but the way the *existing*
> information had to be handled exactly the other way round.
>
> Even worse, some manufacturer adopted C37.118 but still called it "1344
> extensions", which resulted in a huge mess.
For sure a big mess. I know about this line of history. Today we should
consult C37.118.1 and not the older C37.118, but as I recall there is no
difference on the IRIG-encoding between the two.
> But guess who had to take care about customers and other users, explain
> to them why "1344" suddenly worked differently than before, and how they
> could solve their problems?
I hear your pain, for sure.
> It was *not* the guys who had written and changed the specs. It was the
> guys that provide devices and services who had to provide support for
> these "corner cases".
>
> I really wouldn't appreciate if this was repeated with NTP, where just
> another version works completely different than earlier versions, and
> manufacturers and service provider have to take care and teach the users.
>
> As I said before, if you want to make such basic changes to protocol
> that has been long existed and is known to use UTC time stamps, you
> should give a very different name to the beast.

We do, we do call it NTPv5. It's also a version number in the header so
that each packet can be treated according to the protocol it says it is.
This is why the version number is in the protocol in the first place.

When one steps the major version of a protocol, in this context this
means that things will be quite different. Look at IPv4 vs IPv6 for
instance. While they service-wise provide similar service, there is a
number of changes. The header is larger, the fields for source and
destination addresses is larger and thus gets shifted positions of
things in the header. Never the less, we have the ability to handle IPv4
and IPv6 in parallel on the same wires and same machines. Sure, there is
bugs, but those resolve itself over times. Similarly there exists other
protocols such as IGMPv1, IGMPv2 and IGMPv3, and today we battle between
having to use IGMPv3 vs IGMPv2, but not because of bugs, but if one
deems the simpler IGMPv2 sufficient feature-wise or one needs to go to
IGMPv3.

So, this concern of yours is indeed very valid, and I do feel your pain,
but we do have methods to handle it.

Now, one can make the comment that someone could make a NTP client that
does not look on the version number, and keep asking and interpret NTPv5
servers as being NTPv4 replies, something which should only occur if
they falsly sent NTP packets marked as NTPv5. If we make incompatible
formats, as some advocate for, these will not work, but also, these will
be considered buggy. The remedy for that would be that such clients is
directed to NTPv4 servers until upgraded to support proper NTPv5
support. I think that is starting to be fairly thin problem, as it would
require pure deception of those clients to start with. I don't see that
we have prohibited a server of serving NTPv4 clients with NTPv4 replies
and NTPv5 clients with NTPv5 replies. If you can show me how that would
work out incorrectly, I really would appreciate the insight.

So, I really feel your pain with regards to IRIG-B, but I think the
lesson do not apply here, as I think we have sufficient means to avoid
the problem just by using the version field. For each version, it must
be extremely clear how to implement things to do it correctly, that I am
very committed to. It needs to be extremely clear in the full documented
model and is part of the protocol specification.

>
>>>>>>>> Will that satisfy all needs? Maybe not. OK, but will this provide a
>>>>>>>> vehicle for more variants. Seems likely if we have a standard way of
>>>>>>>> augmenting the core timing with additional parameters and users add the
>>>>>>>> mapping.
>>>>>>> That's also fine, but by default it should IMO be possible to provide
>>>>>>> simple clients with time in a way as compatible as possible.
>>>>>> Indeed. I am confident that the client can be very simple and provide
>>>>>> the right time, even with leap seconds occurring.
>>>>> Of course. But the limitation is in the server. Right now, it is
>>>>> sufficient to have a time source like DCF77. If you enforce using of
>>>>> TAI, that's not sufficient anymore.
>>>>>
>>>>> You only need the UTC/TAI offset if your system supports TAI, but there
>>>>> are many applications where this isn't supported, and where it's not
>>>>> even a requirement.
>>>> Actually, your logic is NTPv4 centric.
>>> No, my logic is based on the application and usability, for systems with
>>> high requirements as well as for simply systems, of which there are lots.
>> How then is leap-seconds handled, easily. I fail to see that being done
>> easily.
> As has been pointed out before, you only shift the problem elsewhere but
> don't solve it by using TAI.
I never said anything else? Where is this notion coming from that I was
about to run away from that responsibility, no, quite the opposite, it
becomes a responsibility!
> Even if you do all the computation in TAI, and adjust the TAI system
> clock, which benefit has an application that needs time based on UTC?
It has been stated multiple times, that the core computations becomes
easier with a time-scale which is linear, because you can now avoid
having to handle the consequences of the timestamps not directly
represent an linear progression of time, but needs augmenting, with
extra checks etc.
> How is the kernel improved by this so that applications don't see a time
> step back in CLOCK_REALTIME when a leap second is inserted?

Actually, the issue of kernel in operating system isn't the same as the
core calculatings I talk about.

The processing hurdles related has really nothing to do with the kernel,
and the end product can be delivered to the kernel processing as needed
without that changing.

>
>>>> Now, I tried to make very clear
>>>> it would push requirements onto servers if you would take the suggested
>>>> path. I'm not going to hide that, rather, I want people to understand
>>>> that this would be the logical consequence. However, once that is taken,
>>>> the other parts would become much easier.
>>> At the server side, if you want or need to provide time synchronization
>>> for TAI-based systems and for systems that don't need/use it, in any
>>> case you have to take care to get a timestamp and TAI/UTC offset
>>> *consistently*.
>>>
>>> The only question is whether you put TAI into the base packet and apply
>>> the offset to yield UTC, or vice versa, where IMO the latter is much
>>> more appropriate for simple systems.
>> Except that when you do your core processing, you need that to handle
>> the occurrence of leap-seconds in all it's processing.
> Maybe, but it shouldn't be too hard if you have consistent time stamps,
> and it also works like before on systems that don't support TAI.

Not being too hard etc. is well, sure, but that moves the complexity
there, and it moves the complexity to a very timing sensitive part, and
also bug-prone part. It is exactly this which drives the wish for a
linear time-scale, thus having TAI-like properties.

Having implemented similar enough stuff, I know there is enough of
hurdles in that core time-processing as it is. Adding UTC support ended
up being relatively cheap. Being able to build full IRIG-B with
C37.118.1 support was a reasonable challenge.

>
>> If you use a
>> TAI-like time-base in the core. There is a whole line of checks and
>> balances in that core which just never add to their complexity... in a
>> simple system. It's this which is the actual point of achieving that.
>> But then that will come at a cost. It will have the side-consequence of
>> servers needing to know what to do. Simple clients will shift some of
>> their complexity from core processing to the output adaptations, sure.
>> It is only by looking at all those checks and balances we can make the
>> informed decision.
>>
>>>> If you attempt to go the other route, in which you have a proliferation
>>>> of how many time-scales servers can support, you then push out to
>>>> intermediary and clients nodes to handle the hurdles. There are many
>>>> part of the algorithms we are used to use that will become complex, or
>>>> you would have to push to the users only to choose servers with
>>>> compatible time-scales, which would be even worse as it would deepen the
>>>> division we already seeing.
>>>>
>>>>>>>> Sure, I am advocating for a particular part of the solution space. But I
>>>>>>>> think it is doable without any of the parts becoming cumbersomely
>>>>>>>> complex to test and verify. An example is how PTP has extension fields
>>>>>>>> extended by SMPTE 2059-2 and the various output mappings documented
>>>>>>>> separately in SMPTE 2059-1 to show how the transported parameters
>>>>>>>> generate all the legacy timing things. SMPTE 2110-10 then extends SMPTE
>>>>>>>> 2059-2 and 2059-1 for the application within the SMPTE 2110 transport
>>>>>>>> protocols and timing model of media used there. Anyway, I think it is a
>>>>>>>> fair model that seems to work.
>>>>>>> If a *simple* client needs to do this just to derive UTC (what's
>>>>>>> probably the case for most not telco devices) than it's a wrong approach
>>>>>>> to provide TAI and derive UTC from it.
>>>>>> Well, it may seem so at first, but if we taken on the burden of getting
>>>>>> TAI and TAI-UTC and transport that, the client can be very simple and
>>>>>> provide TAI, UTC, POSIX, LINUX, PTP, GPS time scale replicas using very
>>>>>> little code or complexity. The mappings provided will not require many
>>>>>> pages and it will be easy to implement them all.
>>>>>>
>>>>>> Check your inbox for a separate example.
>>>>> I've seen it. I know conversion between different time scales can be
>>>>> done easily *if* all required information is available. I've also
>>>>> written lots of functions that convert between different time scales.
>>>> Yes, I know you know that field very well. I just wanted to be explicit
>>>> so we where agreeing what we where talking about.
>>>>> AFAIK, there's no current OS that doesn't support UTC, but there are
>>>>> systems that don't support TAI. So why not use the time scale that is
>>>>> most supported, and support other scales if they are supported/required?
>>>> Exactly what do you mean with "support UTC"? Does all OSes you know
>>>> support time as 23:59:60Z? If they don't, they do not fall into "support
>>>> UTC" in my book. Some do support mechanisms that enables user layer to
>>>> print 23:59:60Z, but far from all. Those that does just renumber UTC
>>>> leap second to either 23:59:59 or 00:00:00 with no other indication does
>>>> not support UTC in my book.
>>> OK, then let us call it POSIX time. We all know that you'll never see a
>>> second 60 in the kernel clock because it only counts binary seconds anyway.
>> POSIX time is one such time, yes.
>>> A huge part of the whole problem is that an inserted leap second is
>>> originally defined as a numbering of seconds like 58, 59, 60, 0 in
>>> human-readable format, which is totally unsuitable for timekeeping in an
>>> OS kernel.
>> Agree.
>>> However, you can convert this properly from the binary format to a
>>> second "60" on a display *if* you have some associated status
>>> information available with the timestamps, *and* the conversion routines
>>> in the runtime library evaluate that status.
>> Completely agree.
>>> For example, the API calls for Meinberg PCI cards return seconds in
>>> POSIX format *and* an associated status that tells you the current
>>> timestamps *is* part of the leap second.
>> Which is just what a well designed interface do, and I am not surprised
>> you did it well, rather I expect it without checking the details.
>>> If that status information is not available, you just pulsh the problem
>>> to a different location, but you don't solve it.
>> Which is part of the problem that needs to be solved one way or another.
>> I am very aware of these problems.
>>
>> Now, the trouble is that people confuse POSIX time_t to represent UTC
>> and think that any POSIX time_t like system will do UTC on it's own, and
>> that does not work out if you do not one way or another have that extra
>> information. Also, you need to have heads-up. Different systems enables
>> or requires (max and min time) pre-information about pending
>> leap-second. Some limited have ability to indicate without heads-up. So,
>> if you now needs heads-up, and previous NTPv4 had 1 day heads-up, for
>> NTPv5 we seem to opt for allowing more, you need to know that the
>> TAI-UTC number is to shift.
> The question how long in advance to announce a leap second has also been
> discussed zillion times.
>
> Some folks tend to accept a leap second only at the end of June or
> December, which is current practice from IERS. Have you seen the new
> bulletin C 61, which has just recently been published? This triese to
> avoid accepting "false alarms" e.g. as we have seen from broken GPS
> receivers at the end of September.
>
> Then other folks say that a leap second had to be accepted at the end of
> *any* month, because this could happen in theory, and in worst case.

Actually, reading the ITU-R definition of UTC, which IERS decisions
honours, allow for it at the end of each month but with first preference
on end of UTC June and end of UTC December, second preference on end of
UTC Mars and end of UTC September, and then for any other month.

I have a GPS receiver which take the upcoming leap second indication
flag of the Motorola OEM GPS receiver to indicate upcoming leap second
at the end of current quarter, thus any of first or second preference
choices. If it would have implemented reading out the meta-data in the
GPS announcement, the actual GPS date of the new leap second would be
the one to use, but they didn't do that. Now this may not sound like a
bug, until the fact that GPS Ground Segment operations then insert the
data for upcoming leap second fairly quickly after IERS Bullentin C has
announced one, so typically more than 5 months in advance. Most
receivers will correctly execute this leapsecond into its outgoing UTC
replica, except this reciever as it will schedule if 3 months in advance
because of this bug. The transport format they chose to use did not
allow for how the groundwork system of IERS, GPS specification
(IS-GPS-200), GPS operation worked. It is exactly because of this I try
to advocate for us to have a system which does not build on our own
assumptions of how that is probably going to be used, but rather
understand that we can expect the full dynamics of this and those
operating it expect us users to comply to it because they have shown
clearly how it could potentially be used and want to maintain their
freedom to use the full dynamics to operate as needed to go forward,
thus you can expect assumptions to be broken. This is why you need to be
able to indicate in good time advance, and tell WHEN the event occurs.
When that has been designed such, and respected, it has worked.

Also, if you want to build a receiver that is robust, you can observe
the announcement early, you can observe it from the full set of
satellites you see, for over 5 months before the actual event. You can
store it in non-volatile memory such that a power-cycle or full restart
does not make you to forget it. Naturally, you should be able to whipe
it out if incorrect, and that is also part of the DHS Common framework
for PNT resilience require, at resilience level 1.

> Also you want to pass the leap second warning down to a chain of clients
> as soon as possible. On the other hand, a Linux kernel must not receive
> it more than 24 hours in advance, so even if you implement the
> transmission in the protocol in a more sophisticated way,
> implementations have to take care how to evaluate this.
Of course. Exactly how the Linux kernel implement things right now and
in the future is two different things. Actually, as long as we feel it
reasonable to implement, we don't care.
>
>> For NTP on wire protocol we have to recall,
>> the leap-second can occur between a pair of packets, so we need to know
>> in advance.
> You *do* know in advance, if the server has the information available
> early enough.
Yes. That's my point. You need to know in advance. There is interesting
consequences out of that.
>
>> So, if you need to know of the shift, you have almost the
>> same problem as knowing which TAI-UTC you have and when that will step.
> Again, in most cases where the OS supports leap seconds, the algorithm
> just passes the warning down to the kernel, and as long as the OSs are
> not fixed / improved, the problems seen by *applications* that use the
> system time are the same, regardless whether the on-wire protocoluses
> TAI or UTC-like POSIX.

This is a discussion I feel is very much confused by mistaking "core
calculations" with that of kernel. With "core calculations" I mean the
overall basic processing of time-stamps, accepting packets, producing
the time error and round-trip, conclude how this relates to the steered
clock and then produce steering. That's the core calculations. As you go
over to steer a clock, you may or may not directly hit on what timescale
that the steered clock operates in, and any needed time-scale mappings,
which is a concern being there all the time.

We do not let kernel implementation details control how the on-the-wire
protocol looks, we just must sure they fit with reasonable
implementation burden. Within that reasonable implementation burden is
both the core-calculations and the adaptations.

>
>> The whole problem of leap-second handling to support UTC thus require
>> heads-up already at the server, and for all the intermediary nodes all
>> the way to the client, so that the client can do the right thing for
>> it's kernel and user process when the leap second do occur. There is no
>> way around that part.
> Agreed, but this already works pretty well in NTPv4. If at the top of
> the chain a wrong leap second announcement is inserted, you have a
> problem. If at the system clock level the leap second is handled in a
> strange way, you also have a problem, though a different one.
>
> None of these can be solved by just using TAI timestamps on the wire.

If people felt it was working "pretty well" in NTPv4, why are they then
very interested in getting a linear time-scale for timestamp, the only
real reason I know of is because it reduces the processing in one part
which is already a bit messy.

Interestingly enough, there is ways to mitigate incorrect leap-second
errors. Using the same basic rule of using at least three servers, you
can do majority voting. You can do that way before the event, as you
have more than 5 months for most servers. Done correctly, you can even
provide heads up that a server is problematic with it's announcement in
good time before, and mitigate that servers knowledge in good time
before the event. The needed check will not be advanced if you want to,
and it can be a standard reporting outlier that you can have months of
time to fix.

>
>> If you say that only non-leapsecond time-scales is supported, you can do
>> much simpler things. Then again, you have ended up ruling out the proper
>> UTC support. So either we make these properties split fully, or we fuse
>> them together in such way we agree it will be the simplest and most
>> robust way, and then certain UTC specific consequences will end up
>> defining a number of properties.
>>
>> Then, there is a strong wish to keep the core time-processing as clean
>> as possible, and the properties of a TAI-like timestamp format is
>> attractive. That will create challenges for servers and clients.
> That's perfectly fine with me, but IMO it shouldn't be called NTP then.
> For example, if you called it ETP, you could say:
>
> "Unlike NTP, which assumes UTC timestamps, the ETP protocol works with
> TAI, just similar to PTP."
>
> That would avoid a huge amount of confusion.
Again, we do that when we say it's NTPv5.
>
>> However, making all servers provide compatible time has additional
>> advantages as one wants to entertain the capability of combining
>> responses from servers.
> Of course, if you can determine UTC/POSIX or TAI from the server
> response, everything is fine, regardless whether you can consistently
> derive UTC from TAI, or vice versa. ;-)
>
> In a previous email I gave an example how you can consistently derive
> TAI from UTC, if you need TAI.
Which is just the same process, in reverse. I've used that one for many
years now.
>
>> Now, traditionally that have not been an issue.
>> However, we see servers doing UT1 and others doing smeared leap-seconds
>> in parallel with thus providing UTC based time (more or less good,
>> depending on implementation, driver etc). In fact, people wish there to
>> be a more open bag of time-scales. That will end up driving further
>> division rather than improving strength.
> Hm, I disagree here. If you can properly determine determine UTC and TAI
> from the core protocol and algorithms, there should be no problem to
> support other time scales for special cases as well e.g. by using
> associated extension fields.
You confuse the case of there being the timestamp time-scale being
NTPv4, leapsecond-smeared UTC, UT1 etc. from when you have a common core
time-scale and provide mappings to the others. I already advocate for
the later, but you and I disagree on the common core time-scale being
that of NTPv4 or a TAI-based time-scale.
>
>> So, we either have to take favorite topics of the table, or find a way
>> to fuse them into a system that achieves all he goals, and with the wish
>> list I have seen, using TAI-like base and provide mappings in and out is
>> the only one I can forsee to be fruitful, and even that will come with
>> it's costs, where the servers will hurt most, but I am greatly
>> considering that being worth the cost. Yes, a simple client will need to
>> do a little more to convert time, but that will be the small pain
>> compared to the server side, which I still consider worth having.
> And what about simple servers that don't support TAI, or don't have a
> source for the TAI/UTC offset?

If you go back in the mail-thread, you will find a discussion on how
that can be resolved. There is multiple ways that servers can be
supplied with the TAI-UTC difference such that they can resolve the
issue. The whole point is that if you support UTC, you need to support
leap-seconds, and then you need to have advanced knowledge of them in
order to get correct time, regardless of using NTPv4 or other
core-timescale. You end up already needing to resolve quite a bit of the
problem.

Now, if you is in the realm of doing UTC with leap-seconds as NTP-time,
then you need heads up information. Getting the TAI-UTC difference puts
a bit additional requirements on the server.

If you are in the realm of doing leap-second smeared UTCish as NTP-time,
then you need heads up information at the server.

If you are in the realm of doing TAI-based with TAI-UTC leap seconds as
NTP-time, then you needs heads up information, you need to acquire TAI-UTC.

So, we end up having similar enough requirements on the server side and
the source you derive timing from. Even for a simple server.

I think it is doable either way, but the differences isn't as dramatic
as it may first look. However, as we consider the various systems,
consequences on implementation become different.

Cheers,
Magnus