Re: [Ecrit] planned-changes: two questions

[Brian Rosen <[e-mail]>]

Dan’s proposal doesn’t use IDs.  He returns partial civic addresses.

> On Sep 13, 2021, at 6:12 PM, Randall Gellens <[e-mail]> wrote:
> 
> Fitting his concept into the draft leads us to retain the concept of the server returning an ID with a query result; the change is that instead of a notification URI we have a new query. I suppose this new query (e.g., "plannedChangesRequest") could either return a list of all IDs potentially invalidated with an "asOf" date, or the query could contain a set of IDs and the response can indicate the "asOf" date for each queried ID.
> 
> --Randall
> 
> On 13 Sep 2021, at 14:45, Brian Rosen wrote:
> 
> Well, since there is no planned changes, it’s suggested that LISs revalidate periodically.  I think 30 days is the suggested frequency.
> A user configured address is validated when they enter it.  That doesn’t work well now, but newer systems using LoST will make that immediate.  The systems usually don’t allow an invalid location to be configured.  I never thought that was a good idea, but that’s what I’ve seen.  They don’t initiate service, or keep the old address until it’s validated.
> 
> Yes. I think you have described Dan’s proposal correctly.  He didn’t have IDs, the system delivers addresses and AsOf dates.  He had the notion of a partial civic address, which may make the change list much smaller.
> 
> Brian
> 
>> On Sep 13, 2021, at 5:27 PM, Randall Gellens <[e-mail] <mailto:[e-mail]>> wrote:
>> 
>> Do LISs frequently revalidate addresses? An address that's user configured is likely to be wrong (not where the user is at the time of a call) but still valid.
>> 
>> If I understand Dan's suggestion and how it would fit into the draft, the LoST server returns an ID in the response, as in the current draft, but instead of the client providing a URI which the server stores and later sends a POST to, we define a new LoST query (e.g., "plannedChangesRequest"), and in the response to that, the server includes sets of IDs and "asOf" dates. Is that it?
>> 
>> --Randall
>> 
>> On 13 Sep 2021, at 5:30, Brian Rosen wrote:
>> 
>> The kind of thing I’ve seen many times is an annexation.  It’s announced months, sometimes years in advance.  While there are clear boundaries, they very often don’t match postal boundaries or the existing “unincorporated community name”, A4.  It takes weeks to months of prep to figure out which entries will change and which stay the same.  Of course a good reason it takes that long is that at present, there is nothing like LoST Planned Changes, so it’s got a whole lot of manual work.  I am aware that sometimes, things really do change where you don’t find out more than 24 hours ahead.  Usually, that’s poor planning, rather than an actual short fuse change, but it happens, and has to be dealt with.
>> 
>> Just polling is a really poor answer IMO, because you have to revalidate the entire dataset to find the entries that changed.  I’ve built many systems with TTL where the TTL was gradually reduced around a change so that the entries that changed had short TTLs and the entries that didn’t had longer ones.  That only works if you know far enough in advance that a change is coming.  That’s generally true for the events that planned-changes is designed for, but not always.  If your norm is 90 day TTL, and something comes up where you find out 30 or 60 days in advance, then that idea won’t work.
>> 
>> Dan’s system fixes that, by having a poll for changes.  He publishes a list of entries that will change.  That can be polled frequently, because it’s small.  If I understand the “partial civic” idea, it can often be very small.
>> 
>> I still prefer push, because it actually fits the problem: the server wants to tell the client that something is changing.  But I’m willing to do what Dan suggests.  It’s not my favorite, but it will work.  Just using TTL won’t, in my opinion.  Either you have to do way too frequent full revalidations, or you miss events.
>> 
>> I’ve never liked the fact that LoST servers have 99.99% of transactions that are mostly worthless - revalidations of things that didn’t change.  
>> 
>> But as editor, I will write up whatever the work group consensus is.
>> 
>> Brian
>> 
>>> On Sep 10, 2021, at 4:11 PM, Jeff Martin <[e-mail] <mailto:[e-mail]>> wrote:
>>> 
>>> Brian wrote:
>>> > [the client] not knowing well in advance [of a server change] is, I think, unacceptable
>>>  
>>> In my outlined suggestion, I ended with:
>>> The data in the [server] issued set could be further extended to convey upcoming potential impacts to ids that have not yet been impacted.
>>>  
>>> The server's issued set(s) could contain 'invalidAsOf' indicators with the locationIds, similar to the draft's server-pushed <locationInvalidated invalidAsOf="..." ...>.   So if a server knows "well in advance", say 24 hours, as long as the clients pull within 24 hours the clients would also know well in advance.   My outlined suggestion also includes server-provided TTL in server responses to give hints to clients about polling frequency.
>>>  
>>> Balancing the tradeoffs of polling vs uri+push requires answering this question:  What will be the typical and/or exceptional lead-times for servers operators to know about upcoming changes?
>>>  
>>> If it's common that server operators themselves often identify changes with less than one-hour notice, then the complexity of client-registered-uri with server-push is more justified.
>>>  
>>> But if it's common that server operators usually identify changes with one day or more notice, and one-hour notice is very uncommon, then the simplicity of polling is more justified.
>>>  
>>> IMO based on experience at my company, days of lead time (not hours) is typical for server operators dealing with "planned changes" that affect validity.  So the simplicity of polling with responses that include  'invalidAsOf' is better than the complexity of uri+push, whose benefits would rarely be realized in an environment dominated by days  of lead time.
>>>  
>>> /Jeff/
>>>  
>>>  
>>> From: Brian Rosen <[e-mail] <mailto:[e-mail]>> 
>>> Sent: Friday, September 10, 2021 10:30 AM
>>> To: Dan Banks <dbanks=[e-mail] <mailto:dbanks=[e-mail]>>
>>> Cc: Jeff Martin <[e-mail] <mailto:[e-mail]>>; [e-mail] <mailto:[e-mail]>
>>> Subject: Re: [Ecrit] planned-changes: two questions
>>> 
>>> 
>>> Minutes? 
>>>  
>>> So the clients of this service poll for changes every few minutes?
>>>  
>>> What is a “partial civic address”?  Example?  Something like an A1/A2/A4 but no street name/number, meaning any address in this unincorporated community name?
>>>  
>>> ISTM that a planned change quite often can’t be fixed in a way that code could do it.  So, we still need to advertise the change well in advance of the change, and have the ability to do the AsOf validation.  Changing the way the client discovers that a change is coming is one thing, not knowing well in advance is, I think, unacceptable. 
>>>  
>>> Brian
>>>  
>>>  
>>>  
>>> 
>>> 
>>> ___________________________________________________________________
>>> On Sep 10, 2021, at 12:13 PM, Dan Banks <dbanks=[e-mail] <mailto:dbanks=[e-mail]>> wrote:
>>>  
>>> I am opposed to asking LoST servers to store URIs as well.  I believe it significantly overcomplicates the solution and is the biggest reason why I have not supported the planned changes draft.
>>>  
>>> Several years ago we implemented a mechanism to allow a LIS to know that it needs to revalidate.  Roughly described, that mechanism consists of a web API that can be queried for changes after a particular time.  The response is a summary of changes in terms of partial civic addresses that allow the LIS to identify records that are likely to be affected, or possibly that there have been so many changes that a full revalidation is recommended.  We’ve made the API discoverable via U-NAPTR using the same app string as a given LoST server.
>>>  
>>> Yes, the LIS has to poll the various LoST servers and ask if there have been any changes.  That is a cheap operation to implement, and the practical difference between knowing the very instant a change is published versus finding out within a few minutes is not significant in any way.
>>>  
>>> Dan
>>>  
>>> From: Ecrit <[e-mail] <mailto:[e-mail]>> On Behalf Of Jeff Martin
>>> Sent: Thursday, September 9, 2021 4:09 PM
>>> To: [e-mail] <mailto:[e-mail]>
>>> Subject: Re: [Ecrit] planned-changes: two questions
>>>  
>>> Asking a server to store and later user a client-provided uri opens the door for abuse by malicious actors.   Which got me to thinking, what if we removed the client-provided uri entirely?  If no uris then no risk of abuse.
>>>  
>>> Keep the new optional <plannedChange asOf="..."> but without 'uri' in the client's <findServiceRequest>.  Keep the new optional <ttl> in the server's <findServiceResponse> as a hint to clients that may want to initiate revalidation from the client side.  Keep the new optional server "unique location ID" in the server response as discussed on this list (but not yet in published draft), which the client could choose to associate with the client's internal records.  
>>>  
>>> The server periodically (server policy) issues a set of "potentially impacted" ids, where "potentially impacted" has already been discussed on this list. Each set includes: the ids that have been "potentially impacted" since the previous set was issued; the time-range covered; and a TTL hint on how long until the server is likely to issue the next list.  The server keeps the sets going back some interval defined by server policy.
>>>  
>>> Client sends a "discovery" query to server, where server response is: the url(s) of the most recent and past set(s); and a TTL hint on how long until next list likely to be released.  Client then queries any of these urls, and server responds with the set that includes: the ids that have been "potentially impacted" since the previous set was issued; the time-range covered; and a TTL hint on how long until the server is likely to issue the next list.
>>> 
>>> For clients that want to be highly proactive for changes, the client can store the "unique location id" from <findServiceResponse> and associate to the client's internal records.  The client then periodically queries the server for the set(s) of impacted IDs, and client gets a copy of each set. The client uses these sets of impacted ids from the servers list to check the clients internal records, and client can then revalidate those.
>>>  
>>> No more client-provided uris entirely removing the possibility of abuse by malicious actors. 
>>>  
>>> The data in the issued set could be further extended to convey upcoming potential impacts to ids that have not yet been impacted.
>>>  
>>>  
>>> /Jeff/
>>>  
>>> From: Ecrit <[e-mail] <mailto:[e-mail]>> On Behalf Of Caron, Guy
>>> Sent: Thursday, September 9, 2021 7:52 AM
>>> To: Brian Rosen <[e-mail] <mailto:[e-mail]>>; Randall Gellens <[e-mail] <mailto:[e-mail]>>
>>> Cc: ECRIT <[e-mail] <mailto:[e-mail]>>
>>> Subject: Re: [Ecrit] planned-changes: two questions
>>>  
>>> I’m aligned with Brian on the proposal to embed a reverse HTTP transaction within an existing one. I think this is asking for trouble.
>>>  
>>> Guy
>>>  
>>> De : Brian Rosen <[e-mail] <mailto:[e-mail]>> 
>>> Envoyé : 8 septembre 2021 19:32
>>> À : Randall Gellens <[e-mail] <mailto:[e-mail]>>
>>> Cc : Caron, Guy <[e-mail] <mailto:[e-mail]>>; ECRIT <[e-mail] <mailto:[e-mail]>>
>>> Objet : [EXT]Re: [Ecrit] planned-changes: two questions
>>>  
>>> Inline.  I think we need other opinions.  Certainly, we don’t have rough consensus.
>>>  
>>>  
>>> 
>>> ________________________________________________________
>>> On Sep 8, 2021, at 5:45 PM, Randall Gellens <[e-mail] <mailto:[e-mail]>> wrote:
>>>  
>>> Why is a subsequent transaction better than a parallel one? The subsequent transaction model delays knowledge of the state.
>>> If the client doesn't see an immediate POST, it doesn't know if there's been an error or not.
>>> Yes it does.  If it doesn’t see an immediate POST, then there was a problem and it has to try again.
>>> 
>>> If the client sees a POST and sends an ID, it doesn't know if the server received it and stored the URI.
>>> Technically true, but vanishingly small issue.  You had a complete HTTPS transaction, but somehow the client isn’t sure the server got the ID. 
>>> 
>>> If the server tries a POST but gets an error, it may retry but meanwhile the client may retry the query, since it didn't get a POST.
>>> That's always going to be something the server has to deal with: re-enrollment of the URI.  The client gets a normal response, and that tells it the URI was accepted.
>>> 
>>> Should a client retain IDs if it hasn't seen a POST?
>>> No, lack of POST means URI is not enrolled
>>> 
>>> Also, requiring that the client send an ID seems like it may constrain client architectures since any part of the client system that receives a POST must know which IDs are pending.
>>> I don’t agree.  This is a pretty normal “I send you a magic number and you have to return it to me” thing.  There are no “pending IDs”.  This is the first ID, so you need to return it, and only it, but it’s the only ID the client has at this point.
>>> 
>>> What is the objection to a parallel POST?
>>> I was taught that you don’t embed an HTTP transaction in another with the same partner because things go wrong with timeouts and logic.  It’s not a parallel POST, it’s an HTTPS transaction one way that encloses another HTTPS transaction the other way.  You are dependent on the inner one completing before the outer one times out or has some other issue.  I just think that’s dangerous.  I will admit that I was taught this quite a long time ago (before HTTPS for sure) and things could have changed, but I can’t recall an API that does that.  Also note that keep alive works exactly the same way as the initial POST.  In your proposal, those are different.  That’s not much, but it’s some code.  The “test” transaction tells you that the mechanism works and your URI is being retained, whether that happens the first time, or a year later.
>>> 
>>> --Randall
>>> On 8 Sep 2021, at 14:22, Brian Rosen wrote:
>>> I don’t think there is any fragility in the URI setup.  For a new URI, the client requests the server to keep it in a findService.  That transaction concludes by sending an ID.  The server immediately sends the test notification, to which the client responds with the ID.  At that point both sides know the URI was accepted and the client is valid.  That may be repeated periodically so both sides know the other is happy with the arrangements.  If the client doesn’t get the test notification, it knows the URI isn’t accepted.  If the server doesn’t see the ID, it’s knows that was a bogus request, or there is some other problem, and it shouldn’t save the URI.
>>>  
>>> I proposed a null ID as the test transaction flag.  That would be a piece of XML with the right namespace, the right element name, but no value.  I think that is adequate and meets Guy’s minimal mechanism criteria.  The same XML is always sent.  It either has a set of IDs or it’s empty.
>>>  
>>> Brian
>>>  
>>> 
>>> On Sep 8, 2021, at 5:11 PM, Randall Gellens <[e-mail] <mailto:[e-mail]>> wrote:
>>>  
>>> The proposed mechanism seems fragile. A client has no way to know if its request to store a URI was accepted. That makes it hard to debug. If there are errors of any kind when the server initially verifies a URI, the client has no idea. Even if the client repeats the transaction, the server will silently discard the URI. To me, that's asking for problems.
>>> In my proposal, when the client requests to be notified and provides a URI, the server immediately does a parallel POST to that URI. If it gets a 202 Accepted response to the POST (or we can choose a different value), it stores the URI and returns the query result. If it gets a different response, it does not store the URI and returns the query result with a uriNotStored warning. If a client is trying to get the server to launch an attack on a third party entity, that entity will likely not support the URI in the first place, or will likely return an error response. If you want additional verification, have the server include the queried location in its test POST. That way, if the client maliciously sent a URI of a third party LIS, that LIS will know it does not have a query with that location outstanding.
>>> Doing a parallel POST from the server to the client is one small transaction; this mechanism provides immediate feedback to both client and server. If there are DNS failures or network congestion or whatever, the server returns uriNotStored and the client can try again later.
>>> As for multiple IDs in a single notification POST, having the client specify in the initial request the maximum it is prepared to accept seems reasonable. We can require that clients support a minimum number. If a server has more than the maximum, it sends them in multiple POSTs, with whatever separation in time it chooses. A server can include fewer IDs in a POST than the client's maximum.
>>> We can have a 'test' notification value that is used both for the initial verification test and for periodic keep-alive tests.
>>> --Randall
>>> On 8 Sep 2021, at 12:17, Brian Rosen wrote:
>>> Okay, I think the three of us are converging,  Here is a restatement of your description:
>>> 1) In a validation query, a Client can request to be notified when the proffered LI should be revalidated, and provides a URI to send the notifications to;
>>> 2) In the validation response, the Server provides an ID that the Client associates with the LI it just validated.   The server may silently ignore repeated requests to store a URI where the test in 4 below fails.
>>> 3) Immediately thereafter, if the URI is new to the Server, the Server sends a ‘test’ notification to the URI, with an empty ID.
>>> 4) The recipient at the URI is expected to respond with the ID provided in step 2. If it does, the Server stores the URI for future notifications.  If it does not, the server ignores the request to store the URI. 
>>> 5) Some time after, the Server notifies the Client of an upcoming planned change by sending a notification to the successfully tested URI with the location ID;
>>> 6) The client revalidates each LI in its database that matches the ID as of the date of the planned change. If no ID matches, it is a no-op at the Client. Revalidations may also result in no-op at the Client.
>>> 7) LIs at the Client that are invalidated by the planned change are modified in its database to be valid (which probably mean another revalidation cycle) with an effective date set to <revalidateAsoF> value.
>>> 8) The Server may send ’test’ notifications to the URI without any ID as a form of “keep-alive”.  Any ID provided by the Server to the client may be used as the response to the test transaction
>>>  
>>>  
>>> There has been a discussion of sending more than one ID in a transaction.  I think that is a decent idea, but I worry about how big that could be.  Either we put a hard limit in the text or have something in the response to the test transaction that specifies a size limit for that client.
>>>  
>>> Brian
>>>  
>>> 
>>> On Sep 7, 2021, at 8:31 PM, Caron, Guy <[e-mail] <mailto:[e-mail]>> wrote:
>>>  
>>> 1) In a validation query, a Client can request to be notified when the proffered LI should be revalidated, and provides a URI to send the notifications to;
>>> 2) In the validation response, the Server provides an ID that the Client associates with the LI it just validated;
>>> 3) Immediately thereafter, the Server sends a ‘test’ notification to the URI, without any ID;\
>>> [br[For every new ID?  Or just once?  I wanted this to be a one time registration.
>>> [GC] Just once per offered URI.
>>>  
>>> 
>>> 4) The recipient at the URI is expected to respond with the ID provided in step 2. If it does, the Server stores the URI for future notifications. If it does not, the Server [let’s pick one: reject silently the URI and block the Client permanently/provides a ‘uriNotStored’ warning response to the URI {not compatible with the current proposal to test outside of LoST}/reject silently the URI and block the Client temporarily/other?];
>>> [br]I don’t think an explicit failure is a problem.  The LoST server can limit retries if it needs to.
>>> [GC] Ok for HTTP failures but what I’m talking about is when it fails to return the ID, like in your DoS example.
>>>  
>>>  
>>> 
>>> 5) Some time after, the Server notifies the Client of an upcoming planned change by sending a notification to the successfully tested URI with the location ID;
>>> 6) The client revalidates each LI in its database that matches the ID as of the date of the planned change. If no ID matches, it is a no-op at the Client. Revalidations may also result in no-op at the Client.
>>> 7) LIs at the Client that are invalidated by the planned change are modified in its database to be valid (which probably mean another revalidation cycle) with an effective date set to <revalidateAsoF> value.
>>> [br]I wanted periodic keep alives.  How would that work?
>>> [GC] You mean at step 3? As I mentioned below, I was wondering about the necessity for periodic tests. If the group is convinced it is needed, the Server can simply redo step 3 and the Client can respond with one or many IDs it has from that Server. 
>>>  
>>>  
>>> NOTICE TO RECIPIENT: This email, including attachments, may contain information which is confidential, proprietary, attorney-client privileged and / or controlled under U.S. export laws and regulations and may be restricted from disclosure by applicable State and Federal law. Nothing in this email shall create any legal binding agreement between the parties unless expressly stated herein and provided by an authorized representative of Comtech Telecommunications Corp. or its subsidiaries. If you are not the intended recipient of this message, be advised that any dissemination, distribution, or use of the contents of this message is strictly prohibited. If you received this message in error, please notify us immediately by return email and permanently delete all copies of the original email and any attached documentation from any computer or other media.
>>> _______________________________________________
>>> Ecrit mailing list
>>> [e-mail] <mailto:[e-mail]>
>>> https://www.ietf.org/mailman/listinfo/ecrit <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fecrit&data=04%7C01%7Cjeff.martin%40comtechtel.com%7Cb6354e9963b1469a1a1808d9747845af%7Ca9a26e696ae040c1bd801ca6cc677828%7C0%7C0%7C637668882191609050%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NDkyikK2aPrzaE826qPjFrrlsn1WtrgpqjKx3KBKnQQ%3D&reserved=0>
>> _______________________________________________
>> Ecrit mailing list
>> [e-mail] <mailto:[e-mail]>
>> https://www.ietf.org/mailman/listinfo/ecrit <https://www.ietf.org/mailman/listinfo/ecrit>