Re: [Ntp] WGLC: draft-ietf-ntp-interleaved-modes

[Harlan Stenn <[e-mail]>]

On 12/10/18 8:05 AM, Miroslav Lichvar wrote:
> On Fri, Dec 07, 2018 at 01:36:45PM -0800, Harlan Stenn wrote:
>> While I am supportive of the goals of this document, I am opposed to its
>> acceptance as a standards-track document with the document's current
>> language.
> 
> Thanks for the comments.
> 
>> 1: Requiring an interleaved client/server mode places an unacceptable
>> burden on servers that have a lot of client traffic.  Specifically,
>> requiring a server to save date is too often Bad/onerous.
> 
> The document says "The server SHOULD discard old timestamps to limit
> the amount of memory needed to support clients using the interleaved
> mode." and "The server MAY always respond in the basic mode."

I'm not talking about old timestamps.

I'm talking about a case where there are LOTS of current clients.

The client/server portion of this proposal changes the daemon from
stateless to stateful.

> It can use as much or as little memory for interleaved clients as is
> available or configured. It can drop all state at any time. It can
> enable the support only for some specific addresses. That's perfectly
> fine.

Sure, and that's why I offered my suggested language change below.

> The main use case is synchronization in local networks. There is
> nothing that prevents the use of the interleaved mode over Internet,
> and there are public servers that support it, but it doesn't make much
> of a difference in terms of accuracy, because the jitter and asymmetry
> over Internet is typically orders of magnitude larger than the error
> in transmit timestamp in the basic mode.
> 
>> 2: I would like to see demonstrated testing and test results that show
>> that for interleaved client/server mode, the specification behaves as
>> expected in both hostile and "dirty" environments, and does a thorough
>> job of demonstrating that there are no unintended consequences.  Has the
>> solid engineering been done?
> 
> Yes. An implementation of the protocol has been thoroughly tested with
> replays, packet drops, and other MITM attacks. It's also used in
> production.
> 
> Do you have some specific tests in mind?

Not yet.

> There is a minimal client and server implementation in python. Please
> feel free to test it and report if anything breaks.
> 
> https://github.com/mlichvar/draft-ntp-interleaved-modes

You have this in chronyd then?  Do you have any results on how much of a
difference it makes?

If you're testing this with chrony I would imagine the results would be
slightly different from ntpd, as I don't believe chrony implements the
same internal algorithms and I don't know how significant that would be
to the results.


>> The other is to change the following and then re-evaluate the results:
>>
>> The 4th paragraph of 2. begins with:
>>
>>   A server which supports the interleaved mode needs to save
>>   pairs of local receive and transmit timestamps.
>>
>> I recommend the following instead:
>>
>>   A server which supports interleaved mode for the client needs to save
>>   pairs of local receive and transmit timestamps.
> 
> Ok. If you feel this is not clear from the document, we can say it
> explicitly. Is there a reason you dropped the "the"? There seem to be
> a lot of rules about using definite/indefinite articles in English.
> I'm never sure.

To me, my suggested language is a tighter definition and should make it
easier for implementors (how many will there be?) to do the right thing,
and focus on the relevant issues.

I hadn't noticed I dropped the "the" :)

In this case I would ordinarily not add it, but then again I'm a fan of
the Oxford comma, and we always talk about "Network Time Foundation",
"ISC", etc. instead of "the Network Time Foundation" or "the ISC".

My British friends talk about "going to hospital" while in the US (at
least) we talk about "going to the hospital".

English.  What a language.

>> The 4th paragraph of 2. ends with:
>>
>>   The server MAY separate the timestamps by IP addresses, but
>>   it SHOULD NOT separate them by port numbers, i.e.  clients
>>   are allowed to change their source port between requests.
>>
>> Has this been thoroughly tested?  I am specifically concerned about
>> multiple clients behind a NAT device, where the server will (almost?)
>> certainly see different port numbers for each client.  How well does
>> this work when there are multiple interleaved clients using different
>> ports from the same IP?  Is it sufficient that the server would compare
>> the incoming origin(?) timestamp to identify which (interleave response)
>> transmit timestamp should be sent back?
> 
> The receive and transmit timestamps in server's responses, which are
> copied to the origin timestamp in client requests, are fully under the
> control of the server. A server with a sane clock will not respond
> with a duplicate receive timestamp, so each client using interleaved
> mode sends a unique origin timestamp.
> 
> if the clock is not sane, there may be a duplication and a client
> could get a transmit timestamp of a response that was sent to a
> different client. Both transmit timestamps correspond to the same
> receive timestamp, so the error should be small, but it depends on
> what exactly happened to the server's clock that it produced the same
> timestamp twice. If it was a backward step, it doesn't really matter
> if the response is in basic mode or interleaved mode. All clients
> polling the server around that time would be affected.

I'm not talking about server responses, I'm talking about the server
properly matching stored timestamps when there are an increasing number
of clients behind a single IP.

>> Given that UDP packets may be
>> lost in transit, how does the protocol behave in this case of a lost
>> client request?
> 
> If the client request is lost, nothing surprising will happen. The
> next request will use the same origin timestamp and it will have an
> interleaved response if the server still has the timestamps. If the
> response is lost, the next response may be in basic mode if the server
> is keeping only one pair of timestamps per client.

I'm not sure how "client" is being defined here, given the discussions
about not paying attention to port numbers.

I also need to look at this again to understand your point about "the
next request will use the same origin timestamp ...".

This brings up the question of what happens if a server response is lost
- in that case the server will have sent its interleaved reply and will
switch the client's saved transmit/origin timestamp.  When the client
doesn't get the response, it will then send the next packet with the
previous origin timestamp, the one the server no longer has because it
already replied using that one?

>> How long should the server keep an interleaved transmit
>> timestamp, especially if we are not using the port number to help
>> identify responses? This also goes to the case where an interleaved
>> client "stops" - how long should the server retain the information about
>> the last transmit timestamp for an "association" that is now gone?
> 
> That's up to the implementation. It doesn't have to be a specific
> timeout. It doesn't have to track individual associations. The
> simplest implementation would probably be a single queue with constant
> length for all clients, which would drop the oldest timestamps as new
> timestamps are made. If there is little traffic, the server may keep
> timestamps in memory for a long time.

Yes, and one way to manage this is with a capped queue length, and
another way to manage it is with client limits (if there is a reasonable
way to manage this).  My point goes more towards the belief that there
is benefit to removing old/expired timestamps from the queue sooner
rather than later.  So it may make sense to keep a timestamp for, say, 2
seconds past 2x(poll interval), as we don't know if the client might
decide to increase its poll interval based on whatever it learned from
our most recent response.

>> Should the client/server interleaved mode store the client's poll
>> interval and use that information for this decision?
> 
> An advanced implementation could do that, but I'm not sure if it's
> worth the trouble.

I'm not certain of the costs/risks of keeping old timestamps in the queue.

>> Paragraph 6 of 2. begins with:
>>
>>   When the server receives a request from a client, it SHOULD respond
>>   in the interleaved mode if the following conditions are met:
>>
>> I think this is too strong, and doesn't clearly allow for the situation
>> where a server may only offer interleaved client/server mode to a subset
>> of clients.  I wonder if:
>>
>>   When the server receives a request from a client and is willing to
>>   offer an interleaved client/server association, it SHOULD respond
>>   in the interleaved mode if the following conditions are met:
>>
>> is sufficiently "better".
> 
> Yes, I think that makes it more clear.
> 
>> In the case of a NAT proxy in between the clients and the server, it is
>> possible that two, and possibly more, clients will send packets with the
>> same transmit timestamp.  As the number of clients behind the NAT proxy
>> increases, the possibility of identical transmit timestamps also increases.
> 
> That's not a problem. The server doesn't really care what transmit
> timestamp is in the request. The only thing it does with it is to
> check if it's not equal to the receive timestamp from the same
> request, and use it as the origin timestamp when responding in the
> basic mode.

I will have to study this more.

>> I am also very concerned about this paragraph in 2.:
>>
>>   The client SHOULD NOT update its NTP state when an invalid response
>>   is received to not lose the timestamps which will be needed to
>>   complete a measurement when the following response in the interleaved
>>   mode is received.
>>
>> There are intricate and critical reasons why NTP updates various aspects
>> of its state depending on how far along packet validation and processing
>> occurs.  I would want to see a detailed and specific analysis of the
>> complete effects of the proposed sentence on the protocol machinery.
>> The primary goal is that the protocol be robust.  The secondary goal is
>> that interleave mode works.  I am concerned that the identified sentence
>> above switches these priorities.
> 
> Not at all. The sentence is there to prevent off-path DoS attacks on
> the client. A client (unlike a peer) can safely ignore all packets
> that don't have a valid origin timestamp.
> 
>> As a general question, exactly how much benefit is seen when using
>> client/server interleaved mode?
> 
> It depends on how accurate are the timestamps, how jittery/symmetric
> is the network, and what clock is actually synchronized (system clock
> or HW clock). In a local network with a small number of switches, the
> accuracy of a system clock may improve from about ten microseconds to
> about a hundred of nanoseconds, so about two orders of magnitude.
> 
>> As another general "question", how does this proposal operate in the
>> face of data-minimization?  Should this be stated in the document?
> 
> Good point. It probably should be stated there. The origin timestamp
> is required for the interleaved mode to work, so it must not be set to
> zero.
> 
> Thanks,
> 

-- 
Harlan Stenn <[e-mail]>
http://networktimefoundation.org - be a member!