Re: [Ntp] Antw: Re: Calls for Adoption -- NTP Extension Field drafts -- Four separate drafts

[Harlan Stenn <[e-mail]>]

On 9/3/2019 12:59 AM, Heiko Gerstung wrote:
> Hi Harlan,
> 
> On 02.09.19, 14:31 "Harlan Stenn" <[e-mail]> wrote:
> 
>     Heiko,
>     
>     On 9/2/2019 4:49 AM, Heiko Gerstung wrote:
>     > On 02.09.19, 13:30 "ntp im Auftrag von Harlan Stenn" <[e-mail] im Auftrag von [e-mail]> wrote:
>     >         
>     >     On 9/2/2019 3:59 AM, Heiko Gerstung wrote:
>     >     > Harlan,
>     >     > 
>     >     > I am not sure I understand you correctly. My point was that we can change the base format of an NTP packet because we mark it as version=5. If someone sends a request with version=4, he should get a version=4 response with the v4 packet format. If someone sends a version=5 request, they will get a version=5 response with the new packet format, because the v5 request indicated that the client wants and understands a v5 response.
>     >     > 
>     >     > Maybe we both agree on this one here and just misunderstand each other? ;-)
>     >     
>     >     People are saying that regardless of the version number in the packet,
>     >     one should respond with the version the other side offers, with the data
>     >     that *we* understand.
>     >     
>     >     How can that possibly work if you want to make a v5 packet that changes
>     >     the format and/or content of the base packet?
>     >     
>     >     How can a v4 server that gets a v5 packet possibly respond with correct
>     >     v5 data?
>     > 
>     > I did not say that (or at least I did not mean it). I said that a server supporting both v4 and v5 should always respond with a v4 packet if the request is v4 and responds with a v5 packet it the request was v5. If a v4 only server receives a v5 request, it should not respond (or respond with a v4 packet instead). 
>     
>     I'm saying that if a v3 system gets a v4 (or greater) packet, the v3
>     system should respond with a v3 packet.
>     
>     Similary, a v4 system that gets a v5 (or greater) packet, the v4 system
>     should respond with a v4 packet.
> 
> I am fine with that. No objections whatsoever. 
>     
>     >     > Again, the loop detection algorithm for degree-n-loops would be one benefit from a chain of refids that are passed on from top to down. It would also allow to identify if two or three of my stratum 3 sources get their time from the same stratum 1 source. If we add a flag to the chain that indicates that this InstanceID contributed to the chain using an NTS protected connection, a client could reject any source that is not providing a fully protected chain. Or it prefers a source that can come up with such a chain.
>     >     
>     >     How do you propose getting that refid chain "downstream"?
>     > 
>     > Using an EF. Adding an EF which says "send me the chain" to a request, triggering the upstream server to send its resonse with an EF attached to it that includes the chain.
>     
>     In NTPv4?  I'm seeing incredible resistance to offering OPTIONAL EF
>     proposals (mechanisms) for v4.
> 
> Harlan, you can at any time implement optional EFs in ntpd, they do not need to be standardized in an RFC. If they are picked up by the users, it might happen that they are ending up in a standalone RFC later (or in v5). 

For this to be true we'd need to fix the IANA EF table registry, and
I've been trying to do that for years.

>     I'm seeing people saying they don't see the need for that mechanism
>     because they want to prevent that as a policy choice for others.
> 
> I do not want to prevent this, I want to prevent that we end up adding this and that (optional) feature to v4 and see a ton of v4 implementations, each providing a different subset of all the features. I also do not want to wait another 10 years before we start working on v5, we should shorten the release cycle to a handful of years instead, making more incremental changes to the protocol instead of introducing a new version every 20 years and keep it alive by adding optional features to it during those two decades.

We haven't seen that many implementations yet.

And with what I'm proposing there's a clear path forward for negotiating
what EFs are supported.

Without my proposals, and with no plan in sight to fix the EF table
registry, and if you're correct that optional EFs can be implemented
without an RFC, please tell me how you forsee interoperabililty.

As for the time it takes to produce a new standard, I'm pretty sure you
won't be happy with the results you  get from the path you want to take.

I am also having difficulties understanding how you are advocating for
"more incremental changes to the protocol" in v5 while at the same time
fighting "adding optional features" to v4.

>     >     We've just spent YEARS doing our best to remove meaning from the refid
>     >     field to prevent abuse vectors for a single upstream, and now you want a
>     >     method to clearly identify a chain of upstreams?
>     > 
>     > A unique chain of InstanceID's which does not include any IP addresses or host names is not a way of clearly identifying anyone in that chain. 
>     
>     If you can't map an InstanceID to a system, then how do you detect a loop?
> 
> I think I explained it below.
>     
>     If you *can* map an InstanceID to a system, how do you prevent a bad
>     actor from abusing that "upstream" system to DoS "downstream" systems?
> 
> If a bad actor can abuse an upstream system, all downstream systems are toast anyway, right? But again, the InstanceID is a way to avoid that mapping and at the same time provides a unique ID for each NTP instance. 

I wrote about this in my response too.  The issue goes to making sure
InstanceIDs are unique.  To do this with 100% certainty seems to require
communications between all directly participating parties along with at
least 1 more degree of "neighbors".

This seems to require a lot of traffic, with ongoing updates.

I don't think this passes the cost/benefit analysis.

We could go to UUIDs which we can *assume* are unique.  But we're also
talking about UDP packets for lots of this traffic, which have limited
payload size.

Unless you believe that there should be 2 ongoing communications streams
here, a UDP-based time sync channel and a TCP-based "ancillary
information" channel.

I don't think that passes the cost/benefit analysis, either.

>     >     Indeed, for this chain to work one would need to know all the players in
>     >     the tree between the top of the heap and the "bottom", as  the loop
>     >     detection would need to know the various possible systems in the middle.
>     > 
>     > As far as I can see, you do not need to know anything about the players in the tree. You just get a list of instanceIDs and as long as your own is not included, and there is none appearing twice, you can assume that there is no loop.
>     
>     I floated an idea for a proposal for this probably 20 years ago, and it
>     was shot down.
> 
> You have a great memory. I have no idea what I did 20 years ago to be honest. But just because it was shot down 20 years ago does not mean it is useless, unreasonable or not helpful today. And it definitely does not mean that we should leave it alone because something like this has been discussed in another time and people then decided it is not worth it. 
>     
>     I think it requires negotiation between all connected parties out to
>     more than degree 2, as one must ensure that *none* of the current group
>     (which may have connections to members of outside groups) share a common
>     InstanceID (if I'm using that word correctly).
>     
>     It means that changes in connectivity will cause ripples of
>     re-negotiations of the known participants *and likely the chains of
>     their neighbor groups*.
> 
> This probably depends on the length of the InstanceID. If that ID is created in a randomized way and the random ID is long enough to make it very unlikely that two NTP instances talking to each other end up having the same ID, it can work just fine without any negotiations etc. What happens to IPv6 addresses that are hashed into 4 bytes to create a refid, what is the probability that two systems talking to each other come up with the same value? I know that a chain of 3-4 servers multiplies the chance that any two of them create the same random ID, but if we choose to have a 16byte InstanceID for example, this should be fine. 

We agree that space in an NTP UDP packet is limited, right?

And the most expensive part of this scenario is when a client (which may
be a leaf, and it might be another "collection" of nodes that need to
agree on the uniqueness of InstanceIDs) joins a new group.

This initial joining is going to be the predominant case during startup,
when a clear goal is getting time synchronized quickly.

Do you see where this is going?

I'm not saying we shouldn't discuss this more, I'm just saying there are
some really difficult and intricate tradeoffs here, and I'm pretty sure
it's going to take a fair amount of time and a LOT of testing to get
something that is as robust, predictable and reliable as the protocol
and reference implementation have proven themselves to be.

>     Please imagine this in the face of the NTP Pool.
> 
> The pool typically provides stratum 2 or 3 servers. I think that the pool especially would benefit from being able to identify that my stratum 3  pool servers A,B and C are in fact synchronized to the same stratum 1 server and therefore do not really represent a multi-source solution for my client. At the moment I do not really have a chance to find that out. If I would know, the client could re-resolve the pool dns entries until it finds 4 servers all with different stratum 1 sources.

It's not about the pool servers - it's about the folks who *sync* with
the pool.  Are you suggesting that *all* of the members of the pool
should make sure they are using unique InstanceIDs to make it easier on
folks who connect to the pool?  If so, what would force different pool
operators to cooperate here?

And why do you think it's important to find upstream servers that sync
with different S1 sources?  There's no timing loop involved there, so
what exactly are you testing for?

At some point is it important to identify different S0 sources?  At what
level of granularity do we need to identify "time sources"?  GPS v. some
other GNSS source?  The receiver chip used?  The firmware?  The vendor
of the refclock?

> The worst thing that can happen in the very unlikely event of two servers in a chain accidentally ending up with the same random InstanceID is that the downstream server wrongly detects a loop and rejects this reference. It would be possible to solve this by making it mandatory for an instance to recreate a new random InstanceID if all upstream servers are ruled out due to this loop detection. There are two possibilities here: if this is a false positive and there is no loop, changing my own instanceID will solve it because the upstream server will stick to its InstanceID and the resulting chain does not trigger the loop detection anymore. If it really is a loop, the chain will, after a few polling cycles, again trigger the loop detection because the new InstanceID shows up in the chain. If we add information in the chain about the polling intervals or the age of the InstanceID entries, the instance that recreated its random InstanceID could wait until it is clear that the IntanceID triggering the loop detection remains unchanged. 

What am I missing?  Why would a downstream care if more than one of its
upstreams have the same source?  That is *not* a loop.

>     This might be less horrible if we set up a registry of InstanceID
>     chains, but think hard about this - it's not a problem when we can get
>     time from an authoritative source.
>     
>     The loop problem is only(?) a possibility if we *lose* access to an
>     authoritative source.
>     
>     At least that's my current working premise.
> 
> I would not want something like a registry and I believe we do not need one. I am not sure about the dependency on an authoritative source. Can you explain why you believe that we do not need the loop detection as long as we have an authoritative source?

I was too sparse with my words.

My point is that with a registry (that has a means to authenticate
entries) we don't need to worry about more than 1 machine using the same
InstanceID.

If we don't have a registry with a validation mechanism, one other
option is to query all of the involved nodes (out to at least 1 degree
of their  involved nodes) to make sure InstanceIDs are unique.

Or we can decide that we don't care about InstanceIDs that are not
unique and sometimes collide.

Or we can find other ways to address making sure InstanceIDs are unique.

>     >     When did NTS evolve to provide assertions about the intentions of the
>     >     "middle players"?
>     > 
>     > I did not say anything about intentions. I just stated that it could be useful to verify that the whole chain of clocks talked to each other using NTS. I know that any man-in-the-middle being able to trick its downstream NTS clients into trusting them will be able to fool a client. But that is true in general, i.e. any MITM attack that successfully gets into the chain will be able to spoil the time downstream. 
>     
>     MITM is pretty much a "game over" scenario.
> That's what I said. 
>     
>     I'm very curious about the quality of assertions that we can actually
>     rely on because "the whole chain of clocks talked to each other using
>     NTS" (or any other authentication model).
> 
> Well, we can only trust our direct upstream server. That server, in turn, can only trust its own upstream servers. This is a chain of trust and as I said, if I do not have any access, information or trust in any of the upstream servers of my own upstream server, I cannot assert anything. The intention of this approach/flag is to make sure that a client chooses a chain of NTS protected servers over an alternative that uses unauthenticated NTP. It would also allow the user to tell their clients to not accept any upstream source when its chain does not contain only NTS protected servers.

This "protection" is not unique to NTS.  Private key can be just as
trustworthy, it just doesn't have the ephemeral negotiation that NTS
offers.  There are other choices here as well, that offer equivalent
protection.

And there are limits to what this protection "offers".

I'll note again that Daniel and some others promised that NTS would be
able to offer protection for peer associations as well, and that promise
still hasn't materialized.  This level of protection was a foundational
aspect of the plan Dave Mills submitted years ago.

If folks want robust and reliable enterprise time sources, I submit that
peer mode is an important part of that mechanism.  Peering also goes to
better behavior for loop detection/avoidance as well as for things like
orphan mode.

>     The primary purpose of the REFID was degree-one loop detection.  Once it
>     was there, in an IPv4 world we noticed we could use it for more things.
>      This eventually led to faulty assumptions and problematic behaviors.
>     Unintended consequences, if you will.
> 
> This is why I would like to see that a) the InstanceID is used as the RefID - but make it 16 bytes (or longer) and randomize it. This requires a change in the packet format (which was the start of this discussion). Using the InstanceID to create the chain, offering n-degree loop detection and an indication whether the upstream server relies on NTS/protected information, would be optional and can be implemented using EFs. 

Do you see a way to do this with NTPv4?

We have a 3 bit version field for the packets.  I expect that a "7" in
that field will mean "version 7 or later" and that there will be a
presumably immutable base structure that will give folks access to some
other data value that will identify the actual (>6) version number.

And while NTS may be one way to do this, it should not be the *only* way
to do this.

And to do this with EFs would require folks to allow EFs to be created,
which means they'd have to start understanding that there's a difference
between mechanism and policy, and that just because somebody doesn't
agree with a policy choice doesn't mean the should not allow the
mechanism to decide if that policy choice is used or not.

>     I see the same thing happening above, and with some proposals offered by
>     others.
> 
> I do not see unintended consequences, but that is probably the nature of unintended consequences (if you would see them, they become intended consequences). The fact that this quickly sketched InstanceID thing can lead to unwanted side effects should not hold us back from investigating this and other possible enhancements. The chance of "unintended consequences" is *always* there. You can deal with that by discussions and research. My whole point here is that we should do this. I am not married to the idea of the InstanceID and the chain and n-degree loop detection etc.  - I just believe it would improve NTP and is worthwhile to be investigated further. 

Happy to discuss it.

And sooner or later it needs to be implemented and tested, and holding
off on that until a committee decides it's suitable advanced is, IMO,
far too late.

The other side of this is that unless these changes are carefully and
consciously made, we're going to be living with the "pollution" for a
very long time.

>     > No need to freak out about this, just ideas for v5. Yes, I know that all this is not part of the protocol in its current state. But we are talking about the future version of the protocol and I see use cases for this. 
>     >     
>     > Best Regards,
>     >    Heiko
>     >     
>     > 
>     
>     -- 
>     Harlan Stenn, Network Time Foundation
>     http://nwtime.org - be a Member!
>     
> Thank you for the discussion, Harlan. It helps to point out potential problems as this allows everyone to think about a solution for that problem. 
> 
> Best Regards,
>     Heiko
> 
> 

-- 
Harlan Stenn, Network Time Foundation
http://nwtime.org - be a Member!