Re: [Ecrit] planned-changes: two questions

[Randall Gellens <[e-mail]>]

Fitting his concept into the draft leads us to retain the concept of the 
server returning an ID with a query result; the change is that instead 
of a notification URI we have a new query.  I suppose this new query 
(e.g., "plannedChangesRequest") could either return a list of all IDs 
potentially invalidated with an "asOf" date, or the query could contain 
a set of IDs and the response can indicate the "asOf" date for each 
queried ID.

--Randall

On 13 Sep 2021, at 14:45, Brian Rosen wrote:

> Well, since there is no planned changes, it’s suggested that LISs 
> revalidate periodically.  I think 30 days is the suggested frequency.
> A user configured address is validated when they enter it.  That 
> doesn’t work well now, but newer systems using LoST will make that 
> immediate.  The systems usually don’t allow an invalid location to 
> be configured.  I never thought that was a good idea, but that’s 
> what I’ve seen.  They don’t initiate service, or keep the old 
> address until it’s validated.
>
> Yes. I think you have described Dan’s proposal correctly.  He 
> didn’t have IDs, the system delivers addresses and AsOf dates.  He 
> had the notion of a partial civic address, which may make the change 
> list much smaller.
>
> Brian
>
>> On Sep 13, 2021, at 5:27 PM, Randall Gellens 
>> <[e-mail]> wrote:
>>
>> Do LISs frequently revalidate addresses? An address that's user 
>> configured is likely to be wrong (not where the user is at the time 
>> of a call) but still valid.
>>
>> If I understand Dan's suggestion and how it would fit into the draft, 
>> the LoST server returns an ID in the response, as in the current 
>> draft, but instead of the client providing a URI which the server 
>> stores and later sends a POST to, we define a new LoST query (e.g., 
>> "plannedChangesRequest"), and in the response to that, the server 
>> includes sets of IDs and "asOf" dates. Is that it?
>>
>> --Randall
>>
>> On 13 Sep 2021, at 5:30, Brian Rosen wrote:
>>
>> The kind of thing I’ve seen many times is an annexation.  It’s 
>> announced months, sometimes years in advance.  While there are clear 
>> boundaries, they very often don’t match postal boundaries or the 
>> existing “unincorporated community name”, A4.  It takes weeks to 
>> months of prep to figure out which entries will change and which stay 
>> the same.  Of course a good reason it takes that long is that at 
>> present, there is nothing like LoST Planned Changes, so it’s got a 
>> whole lot of manual work.  I am aware that sometimes, things really 
>> do change where you don’t find out more than 24 hours ahead.  
>> Usually, that’s poor planning, rather than an actual short fuse 
>> change, but it happens, and has to be dealt with.
>>
>> Just polling is a really poor answer IMO, because you have to 
>> revalidate the entire dataset to find the entries that changed.  
>> I’ve built many systems with TTL where the TTL was gradually 
>> reduced around a change so that the entries that changed had short 
>> TTLs and the entries that didn’t had longer ones.  That only works 
>> if you know far enough in advance that a change is coming.  That’s 
>> generally true for the events that planned-changes is designed for, 
>> but not always.  If your norm is 90 day TTL, and something comes up 
>> where you find out 30 or 60 days in advance, then that idea won’t 
>> work.
>>
>> Dan’s system fixes that, by having a poll for changes.  He 
>> publishes a list of entries that will change.  That can be polled 
>> frequently, because it’s small.  If I understand the “partial 
>> civic” idea, it can often be very small.
>>
>> I still prefer push, because it actually fits the problem: the server 
>> wants to tell the client that something is changing.  But I’m 
>> willing to do what Dan suggests.  It’s not my favorite, but it will 
>> work.  Just using TTL won’t, in my opinion.  Either you have to do 
>> way too frequent full revalidations, or you miss events.
>>
>> I’ve never liked the fact that LoST servers have 99.99% of 
>> transactions that are mostly worthless - revalidations of things that 
>> didn’t change.
>>
>> But as editor, I will write up whatever the work group consensus is.
>>
>> Brian
>>
>>> On Sep 10, 2021, at 4:11 PM, Jeff Martin <[e-mail] 
>>> <mailto:[e-mail]>> wrote:
>>>
>>> Brian wrote:
>>>> [the client] not knowing well in advance [of a server change] is, I 
>>>> think, unacceptable
>>>
>>> In my outlined suggestion, I ended with:
>>> The data in the [server] issued set could be further extended to 
>>> convey upcoming potential impacts to ids that have not yet been 
>>> impacted.
>>>
>>> The server's issued set(s) could contain 'invalidAsOf' indicators 
>>> with the locationIds, similar to the draft's server-pushed 
>>> <locationInvalidated invalidAsOf="..." ...>.   So if a server knows 
>>> "well in advance", say 24 hours, as long as the clients pull within 
>>> 24 hours the clients would also know well in advance.   My outlined 
>>> suggestion also includes server-provided TTL in server responses to 
>>> give hints to clients about polling frequency.
>>>
>>> Balancing the tradeoffs of polling vs uri+push requires answering 
>>> this question:  What will be the typical and/or exceptional 
>>> lead-times for servers operators to know about upcoming changes?
>>>
>>> If it's common that server operators themselves often identify 
>>> changes with less than one-hour notice, then the complexity of 
>>> client-registered-uri with server-push is more justified.
>>>
>>> But if it's common that server operators usually identify changes 
>>> with one day or more notice, and one-hour notice is very uncommon, 
>>> then the simplicity of polling is more justified.
>>>
>>> IMO based on experience at my company, days of lead time (not hours) 
>>> is typical for server operators dealing with "planned changes" that 
>>> affect validity.  So the simplicity of polling with responses that 
>>> include  'invalidAsOf' is better than the complexity of uri+push, 
>>> whose benefits would rarely be realized in an environment dominated 
>>> by days  of lead time.
>>>
>>> /Jeff/
>>>
>>>
>>> From: Brian Rosen <[e-mail] <mailto:[e-mail]>>
>>> Sent: Friday, September 10, 2021 10:30 AM
>>> To: Dan Banks <dbanks=[e-mail] 
>>> <mailto:dbanks=[e-mail]>>
>>> Cc: Jeff Martin <[e-mail] 
>>> <mailto:[e-mail]>>; [e-mail] 
>>> <mailto:[e-mail]>
>>> Subject: Re: [Ecrit] planned-changes: two questions
>>>
>>>
>>> Minutes?
>>>
>>> So the clients of this service poll for changes every few minutes?
>>>
>>> What is a “partial civic address”?  Example?  Something like an 
>>> A1/A2/A4 but no street name/number, meaning any address in this 
>>> unincorporated community name?
>>>
>>> ISTM that a planned change quite often can’t be fixed in a way 
>>> that code could do it.  So, we still need to advertise the change 
>>> well in advance of the change, and have the ability to do the AsOf 
>>> validation.  Changing the way the client discovers that a change is 
>>> coming is one thing, not knowing well in advance is, I think, 
>>> unacceptable.
>>>
>>> Brian
>>>
>>>
>>>
>>>
>>>
>>> ___________________________________________________________________
>>> On Sep 10, 2021, at 12:13 PM, Dan Banks 
>>> <dbanks=[e-mail] 
>>> <mailto:dbanks=[e-mail]>> wrote:
>>>
>>> I am opposed to asking LoST servers to store URIs as well.  I 
>>> believe it significantly overcomplicates the solution and is the 
>>> biggest reason why I have not supported the planned changes draft.
>>>
>>> Several years ago we implemented a mechanism to allow a LIS to know 
>>> that it needs to revalidate.  Roughly described, that mechanism 
>>> consists of a web API that can be queried for changes after a 
>>> particular time.  The response is a summary of changes in terms of 
>>> partial civic addresses that allow the LIS to identify records that 
>>> are likely to be affected, or possibly that there have been so many 
>>> changes that a full revalidation is recommended.  We’ve made the 
>>> API discoverable via U-NAPTR using the same app string as a given 
>>> LoST server.
>>>
>>> Yes, the LIS has to poll the various LoST servers and ask if there 
>>> have been any changes.  That is a cheap operation to implement, and 
>>> the practical difference between knowing the very instant a change 
>>> is published versus finding out within a few minutes is not 
>>> significant in any way.
>>>
>>> Dan
>>>
>>> From: Ecrit <[e-mail] <mailto:[e-mail]>> 
>>> On Behalf Of Jeff Martin
>>> Sent: Thursday, September 9, 2021 4:09 PM
>>> To: [e-mail] <mailto:[e-mail]>
>>> Subject: Re: [Ecrit] planned-changes: two questions
>>>
>>> Asking a server to store and later user a client-provided uri opens 
>>> the door for abuse by malicious actors.   Which got me to thinking, 
>>> what if we removed the client-provided uri entirely?  If no uris 
>>> then no risk of abuse.
>>>
>>> Keep the new optional <plannedChange asOf="..."> but without 'uri' 
>>> in the client's <findServiceRequest>.  Keep the new optional <ttl> 
>>> in the server's <findServiceResponse> as a hint to clients that may 
>>> want to initiate revalidation from the client side.  Keep the new 
>>> optional server "unique location ID" in the server response as 
>>> discussed on this list (but not yet in published draft), which the 
>>> client could choose to associate with the client's internal records.
>>>
>>> The server periodically (server policy) issues a set of "potentially 
>>> impacted" ids, where "potentially impacted" has already been 
>>> discussed on this list. Each set includes: the ids that have been 
>>> "potentially impacted" since the previous set was issued; the 
>>> time-range covered; and a TTL hint on how long until the server is 
>>> likely to issue the next list.  The server keeps the sets going back 
>>> some interval defined by server policy.
>>>
>>> Client sends a "discovery" query to server, where server response 
>>> is: the url(s) of the most recent and past set(s); and a TTL hint on 
>>> how long until next list likely to be released.  Client then queries 
>>> any of these urls, and server responds with the set that includes: 
>>> the ids that have been "potentially impacted" since the previous set 
>>> was issued; the time-range covered; and a TTL hint on how long until 
>>> the server is likely to issue the next list.
>>>
>>> For clients that want to be highly proactive for changes, the client 
>>> can store the "unique location id" from <findServiceResponse> and 
>>> associate to the client's internal records.  The client then 
>>> periodically queries the server for the set(s) of impacted IDs, and 
>>> client gets a copy of each set. The client uses these sets of 
>>> impacted ids from the servers list to check the clients internal 
>>> records, and client can then revalidate those.
>>>
>>> No more client-provided uris entirely removing the possibility of 
>>> abuse by malicious actors.
>>>
>>> The data in the issued set could be further extended to convey 
>>> upcoming potential impacts to ids that have not yet been impacted.
>>>
>>>
>>> /Jeff/
>>>
>>> From: Ecrit <[e-mail] <mailto:[e-mail]>> 
>>> On Behalf Of Caron, Guy
>>> Sent: Thursday, September 9, 2021 7:52 AM
>>> To: Brian Rosen <[e-mail] <mailto:[e-mail]>>; 
>>> Randall Gellens <[e-mail] 
>>> <mailto:[e-mail]>>
>>> Cc: ECRIT <[e-mail] <mailto:[e-mail]>>
>>> Subject: Re: [Ecrit] planned-changes: two questions
>>>
>>> I’m aligned with Brian on the proposal to embed a reverse HTTP 
>>> transaction within an existing one. I think this is asking for 
>>> trouble.
>>>
>>> Guy
>>>
>>> De : Brian Rosen <[e-mail] <mailto:[e-mail]>>
>>> Envoyé : 8 septembre 2021 19:32
>>> À : Randall Gellens <[e-mail] 
>>> <mailto:[e-mail]>>
>>> Cc : Caron, Guy <[e-mail] <mailto:[e-mail]>>; ECRIT 
>>> <[e-mail] <mailto:[e-mail]>>
>>> Objet : [EXT]Re: [Ecrit] planned-changes: two questions
>>>
>>> Inline.  I think we need other opinions.  Certainly, we don’t have 
>>> rough consensus.
>>>
>>>
>>>
>>> ________________________________________________________
>>> On Sep 8, 2021, at 5:45 PM, Randall Gellens 
>>> <[e-mail] <mailto:[e-mail]>> 
>>> wrote:
>>>
>>> Why is a subsequent transaction better than a parallel one? The 
>>> subsequent transaction model delays knowledge of the state.
>>> If the client doesn't see an immediate POST, it doesn't know if 
>>> there's been an error or not.
>>> Yes it does.  If it doesn’t see an immediate POST, then there was 
>>> a problem and it has to try again.
>>>
>>> If the client sees a POST and sends an ID, it doesn't know if the 
>>> server received it and stored the URI.
>>> Technically true, but vanishingly small issue.  You had a complete 
>>> HTTPS transaction, but somehow the client isn’t sure the server 
>>> got the ID.
>>>
>>> If the server tries a POST but gets an error, it may retry but 
>>> meanwhile the client may retry the query, since it didn't get a 
>>> POST.
>>> That's always going to be something the server has to deal with: 
>>> re-enrollment of the URI.  The client gets a normal response, and 
>>> that tells it the URI was accepted.
>>>
>>> Should a client retain IDs if it hasn't seen a POST?
>>> No, lack of POST means URI is not enrolled
>>>
>>> Also, requiring that the client send an ID seems like it may 
>>> constrain client architectures since any part of the client system 
>>> that receives a POST must know which IDs are pending.
>>> I don’t agree.  This is a pretty normal “I send you a magic 
>>> number and you have to return it to me” thing.  There are no 
>>> “pending IDs”.  This is the first ID, so you need to return it, 
>>> and only it, but it’s the only ID the client has at this point.
>>>
>>> What is the objection to a parallel POST?
>>> I was taught that you don’t embed an HTTP transaction in another 
>>> with the same partner because things go wrong with timeouts and 
>>> logic.  It’s not a parallel POST, it’s an HTTPS transaction one 
>>> way that encloses another HTTPS transaction the other way.  You are 
>>> dependent on the inner one completing before the outer one times out 
>>> or has some other issue.  I just think that’s dangerous.  I will 
>>> admit that I was taught this quite a long time ago (before HTTPS for 
>>> sure) and things could have changed, but I can’t recall an API 
>>> that does that.  Also note that keep alive works exactly the same 
>>> way as the initial POST.  In your proposal, those are different.  
>>> That’s not much, but it’s some code.  The “test” transaction 
>>> tells you that the mechanism works and your URI is being retained, 
>>> whether that happens the first time, or a year later.
>>>
>>> --Randall
>>> On 8 Sep 2021, at 14:22, Brian Rosen wrote:
>>> I don’t think there is any fragility in the URI setup.  For a new 
>>> URI, the client requests the server to keep it in a findService.  
>>> That transaction concludes by sending an ID.  The server immediately 
>>> sends the test notification, to which the client responds with the 
>>> ID.  At that point both sides know the URI was accepted and the 
>>> client is valid.  That may be repeated periodically so both sides 
>>> know the other is happy with the arrangements.  If the client 
>>> doesn’t get the test notification, it knows the URI isn’t 
>>> accepted.  If the server doesn’t see the ID, it’s knows that was 
>>> a bogus request, or there is some other problem, and it shouldn’t 
>>> save the URI.
>>>
>>> I proposed a null ID as the test transaction flag.  That would be a 
>>> piece of XML with the right namespace, the right element name, but 
>>> no value.  I think that is adequate and meets Guy’s minimal 
>>> mechanism criteria.  The same XML is always sent.  It either has a 
>>> set of IDs or it’s empty.
>>>
>>> Brian
>>>
>>>
>>> On Sep 8, 2021, at 5:11 PM, Randall Gellens 
>>> <[e-mail] <mailto:[e-mail]>> 
>>> wrote:
>>>
>>> The proposed mechanism seems fragile. A client has no way to know if 
>>> its request to store a URI was accepted. That makes it hard to 
>>> debug. If there are errors of any kind when the server initially 
>>> verifies a URI, the client has no idea. Even if the client repeats 
>>> the transaction, the server will silently discard the URI. To me, 
>>> that's asking for problems.
>>> In my proposal, when the client requests to be notified and provides 
>>> a URI, the server immediately does a parallel POST to that URI. If 
>>> it gets a 202 Accepted response to the POST (or we can choose a 
>>> different value), it stores the URI and returns the query result. If 
>>> it gets a different response, it does not store the URI and returns 
>>> the query result with a uriNotStored warning. If a client is trying 
>>> to get the server to launch an attack on a third party entity, that 
>>> entity will likely not support the URI in the first place, or will 
>>> likely return an error response. If you want additional 
>>> verification, have the server include the queried location in its 
>>> test POST. That way, if the client maliciously sent a URI of a third 
>>> party LIS, that LIS will know it does not have a query with that 
>>> location outstanding.
>>> Doing a parallel POST from the server to the client is one small 
>>> transaction; this mechanism provides immediate feedback to both 
>>> client and server. If there are DNS failures or network congestion 
>>> or whatever, the server returns uriNotStored and the client can try 
>>> again later.
>>> As for multiple IDs in a single notification POST, having the client 
>>> specify in the initial request the maximum it is prepared to accept 
>>> seems reasonable. We can require that clients support a minimum 
>>> number. If a server has more than the maximum, it sends them in 
>>> multiple POSTs, with whatever separation in time it chooses. A 
>>> server can include fewer IDs in a POST than the client's maximum.
>>> We can have a 'test' notification value that is used both for the 
>>> initial verification test and for periodic keep-alive tests.
>>> --Randall
>>> On 8 Sep 2021, at 12:17, Brian Rosen wrote:
>>> Okay, I think the three of us are converging,  Here is a restatement 
>>> of your description:
>>> 1) In a validation query, a Client can request to be notified when 
>>> the proffered LI should be revalidated, and provides a URI to send 
>>> the notifications to;
>>> 2) In the validation response, the Server provides an ID that the 
>>> Client associates with the LI it just validated.   The server may 
>>> silently ignore repeated requests to store a URI where the test in 4 
>>> below fails.
>>> 3) Immediately thereafter, if the URI is new to the Server, the 
>>> Server sends a ‘test’ notification to the URI, with an empty ID.
>>> 4) The recipient at the URI is expected to respond with the ID 
>>> provided in step 2. If it does, the Server stores the URI for future 
>>> notifications.  If it does not, the server ignores the request to 
>>> store the URI.
>>> 5) Some time after, the Server notifies the Client of an upcoming 
>>> planned change by sending a notification to the successfully tested 
>>> URI with the location ID;
>>> 6) The client revalidates each LI in its database that matches the 
>>> ID as of the date of the planned change. If no ID matches, it is a 
>>> no-op at the Client. Revalidations may also result in no-op at the 
>>> Client.
>>> 7) LIs at the Client that are invalidated by the planned change are 
>>> modified in its database to be valid (which probably mean another 
>>> revalidation cycle) with an effective date set to <revalidateAsoF> 
>>> value.
>>> 8) The Server may send ’test’ notifications to the URI without 
>>> any ID as a form of “keep-alive”.  Any ID provided by the Server 
>>> to the client may be used as the response to the test transaction
>>>
>>>
>>> There has been a discussion of sending more than one ID in a 
>>> transaction.  I think that is a decent idea, but I worry about how 
>>> big that could be.  Either we put a hard limit in the text or have 
>>> something in the response to the test transaction that specifies a 
>>> size limit for that client.
>>>
>>> Brian
>>>
>>>
>>> On Sep 7, 2021, at 8:31 PM, Caron, Guy <[e-mail] 
>>> <mailto:[e-mail]>> wrote:
>>>
>>> 1) In a validation query, a Client can request to be notified when 
>>> the proffered LI should be revalidated, and provides a URI to send 
>>> the notifications to;
>>> 2) In the validation response, the Server provides an ID that the 
>>> Client associates with the LI it just validated;
>>> 3) Immediately thereafter, the Server sends a ‘test’ 
>>> notification to the URI, without any ID;\
>>> [br[For every new ID?  Or just once?  I wanted this to be a one time 
>>> registration.
>>> [GC] Just once per offered URI.
>>>
>>>
>>> 4) The recipient at the URI is expected to respond with the ID 
>>> provided in step 2. If it does, the Server stores the URI for future 
>>> notifications. If it does not, the Server [let’s pick one: reject 
>>> silently the URI and block the Client permanently/provides a 
>>> ‘uriNotStored’ warning response to the URI {not compatible with 
>>> the current proposal to test outside of LoST}/reject silently the 
>>> URI and block the Client temporarily/other?];
>>> [br]I don’t think an explicit failure is a problem.  The LoST 
>>> server can limit retries if it needs to.
>>> [GC] Ok for HTTP failures but what I’m talking about is when it 
>>> fails to return the ID, like in your DoS example.
>>>
>>>
>>>
>>> 5) Some time after, the Server notifies the Client of an upcoming 
>>> planned change by sending a notification to the successfully tested 
>>> URI with the location ID;
>>> 6) The client revalidates each LI in its database that matches the 
>>> ID as of the date of the planned change. If no ID matches, it is a 
>>> no-op at the Client. Revalidations may also result in no-op at the 
>>> Client.
>>> 7) LIs at the Client that are invalidated by the planned change are 
>>> modified in its database to be valid (which probably mean another 
>>> revalidation cycle) with an effective date set to <revalidateAsoF> 
>>> value.
>>> [br]I wanted periodic keep alives.  How would that work?
>>> [GC] You mean at step 3? As I mentioned below, I was wondering about 
>>> the necessity for periodic tests. If the group is convinced it is 
>>> needed, the Server can simply redo step 3 and the Client can respond 
>>> with one or many IDs it has from that Server.
>>>
>>>
>>> NOTICE TO RECIPIENT: This email, including attachments, may contain 
>>> information which is confidential, proprietary, attorney-client 
>>> privileged and / or controlled under U.S. export laws and 
>>> regulations and may be restricted from disclosure by applicable 
>>> State and Federal law. Nothing in this email shall create any legal 
>>> binding agreement between the parties unless expressly stated herein 
>>> and provided by an authorized representative of Comtech 
>>> Telecommunications Corp. or its subsidiaries. If you are not the 
>>> intended recipient of this message, be advised that any 
>>> dissemination, distribution, or use of the contents of this message 
>>> is strictly prohibited. If you received this message in error, 
>>> please notify us immediately by return email and permanently delete 
>>> all copies of the original email and any attached documentation from 
>>> any computer or other media.
>>> _______________________________________________
>>> Ecrit mailing list
>>> [e-mail] <mailto:[e-mail]>
>>> https://www.ietf.org/mailman/listinfo/ecrit 
>>> <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fecrit&data=04%7C01%7Cjeff.martin%40comtechtel.com%7Cb6354e9963b1469a1a1808d9747845af%7Ca9a26e696ae040c1bd801ca6cc677828%7C0%7C0%7C637668882191609050%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NDkyikK2aPrzaE826qPjFrrlsn1WtrgpqjKx3KBKnQQ%3D&reserved=0>
>> _______________________________________________
>> Ecrit mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/ecrit 
>> <https://www.ietf.org/mailman/listinfo/ecrit>