Re: [Ecrit] planned-changes: two questions

[Randall Gellens <[e-mail]>]

Do LISs frequently revalidate addresses?  An address that's user 
configured is likely to be wrong (not where the user is at the time of a 
call) but still valid.

If I understand Dan's suggestion and how it would fit into the draft, 
the LoST server returns an ID in the response, as in the current draft, 
but instead of the client providing a URI which the server stores and 
later sends a POST to, we define a new LoST query (e.g., 
"plannedChangesRequest"), and in the response to that, the server 
includes sets of IDs and "asOf" dates.  Is that it?

--Randall

On 13 Sep 2021, at 5:30, Brian Rosen wrote:

> The kind of thing I’ve seen many times is an annexation.  It’s 
> announced months, sometimes years in advance.  While there are clear 
> boundaries, they very often don’t match postal boundaries or the 
> existing “unincorporated community name”, A4.  It takes weeks to 
> months of prep to figure out which entries will change and which stay 
> the same.  Of course a good reason it takes that long is that at 
> present, there is nothing like LoST Planned Changes, so it’s got a 
> whole lot of manual work.  I am aware that sometimes, things really do 
> change where you don’t find out more than 24 hours ahead.  Usually, 
> that’s poor planning, rather than an actual short fuse change, but 
> it happens, and has to be dealt with.
>
> Just polling is a really poor answer IMO, because you have to 
> revalidate the entire dataset to find the entries that changed.  
> I’ve built many systems with TTL where the TTL was gradually reduced 
> around a change so that the entries that changed had short TTLs and 
> the entries that didn’t had longer ones.  That only works if you 
> know far enough in advance that a change is coming.  That’s 
> generally true for the events that planned-changes is designed for, 
> but not always.  If your norm is 90 day TTL, and something comes up 
> where you find out 30 or 60 days in advance, then that idea won’t 
> work.
>
> Dan’s system fixes that, by having a poll for changes.  He publishes 
> a list of entries that will change.  That can be polled frequently, 
> because it’s small.  If I understand the “partial civic” idea, 
> it can often be very small.
>
> I still prefer push, because it actually fits the problem: the server 
> wants to tell the client that something is changing.  But I’m 
> willing to do what Dan suggests.  It’s not my favorite, but it will 
> work.  Just using TTL won’t, in my opinion.  Either you have to do 
> way too frequent full revalidations, or you miss events.
>
> I’ve never liked the fact that LoST servers have 99.99% of 
> transactions that are mostly worthless - revalidations of things that 
> didn’t change.
>
> But as editor, I will write up whatever the work group consensus is.
>
> Brian
>
>> On Sep 10, 2021, at 4:11 PM, Jeff Martin <[e-mail]> 
>> wrote:
>>
>> Brian wrote:
>>> [the client] not knowing well in advance [of a server change] is, I 
>>> think, unacceptable
>>
>> In my outlined suggestion, I ended with:
>> The data in the [server] issued set could be further extended to 
>> convey upcoming potential impacts to ids that have not yet been 
>> impacted.
>>
>> The server's issued set(s) could contain 'invalidAsOf' indicators 
>> with the locationIds, similar to the draft's server-pushed 
>> <locationInvalidated invalidAsOf="..." ...>.   So if a server knows 
>> "well in advance", say 24 hours, as long as the clients pull within 
>> 24 hours the clients would also know well in advance.   My outlined 
>> suggestion also includes server-provided TTL in server responses to 
>> give hints to clients about polling frequency.
>>
>> Balancing the tradeoffs of polling vs uri+push requires answering 
>> this question:  What will be the typical and/or exceptional 
>> lead-times for servers operators to know about upcoming changes?
>>
>> If it's common that server operators themselves often identify 
>> changes with less than one-hour notice, then the complexity of 
>> client-registered-uri with server-push is more justified.
>>
>> But if it's common that server operators usually identify changes 
>> with one day or more notice, and one-hour notice is very uncommon, 
>> then the simplicity of polling is more justified.
>>
>> IMO based on experience at my company, days of lead time (not hours) 
>> is typical for server operators dealing with "planned changes" that 
>> affect validity.  So the simplicity of polling with responses that 
>> include  'invalidAsOf' is better than the complexity of uri+push, 
>> whose benefits would rarely be realized in an environment dominated 
>> by days  of lead time.
>>
>> /Jeff/
>>
>>
>> From: Brian Rosen <[e-mail]>
>> Sent: Friday, September 10, 2021 10:30 AM
>> To: Dan Banks <dbanks=[e-mail]>
>> Cc: Jeff Martin <[e-mail]>; [e-mail]
>> Subject: Re: [Ecrit] planned-changes: two questions
>>
>>
>> Minutes?
>>
>> So the clients of this service poll for changes every few minutes?
>>
>> What is a “partial civic address”?  Example?  Something like an 
>> A1/A2/A4 but no street name/number, meaning any address in this 
>> unincorporated community name?
>>
>> ISTM that a planned change quite often can’t be fixed in a way that 
>> code could do it.  So, we still need to advertise the change well in 
>> advance of the change, and have the ability to do the AsOf 
>> validation.  Changing the way the client discovers that a change is 
>> coming is one thing, not knowing well in advance is, I think, 
>> unacceptable.
>>
>> Brian
>>
>>
>>
>>
>>
>> ___________________________________________________________________
>> On Sep 10, 2021, at 12:13 PM, Dan Banks 
>> <dbanks=[e-mail]> wrote:
>>
>> I am opposed to asking LoST servers to store URIs as well.  I believe 
>> it significantly overcomplicates the solution and is the biggest 
>> reason why I have not supported the planned changes draft.
>>
>> Several years ago we implemented a mechanism to allow a LIS to know 
>> that it needs to revalidate.  Roughly described, that mechanism 
>> consists of a web API that can be queried for changes after a 
>> particular time.  The response is a summary of changes in terms of 
>> partial civic addresses that allow the LIS to identify records that 
>> are likely to be affected, or possibly that there have been so many 
>> changes that a full revalidation is recommended.  We’ve made the 
>> API discoverable via U-NAPTR using the same app string as a given 
>> LoST server.
>>
>> Yes, the LIS has to poll the various LoST servers and ask if there 
>> have been any changes.  That is a cheap operation to implement, and 
>> the practical difference between knowing the very instant a change is 
>> published versus finding out within a few minutes is not significant 
>> in any way.
>>
>> Dan
>>
>> From: Ecrit <[e-mail]> On Behalf Of Jeff Martin
>> Sent: Thursday, September 9, 2021 4:09 PM
>> To: [e-mail]
>> Subject: Re: [Ecrit] planned-changes: two questions
>>
>> Asking a server to store and later user a client-provided uri opens 
>> the door for abuse by malicious actors.   Which got me to thinking, 
>> what if we removed the client-provided uri entirely?  If no uris then 
>> no risk of abuse.
>>
>> Keep the new optional <plannedChange asOf="..."> but without 'uri' in 
>> the client's <findServiceRequest>.  Keep the new optional <ttl> in 
>> the server's <findServiceResponse> as a hint to clients that may want 
>> to initiate revalidation from the client side.  Keep the new optional 
>> server "unique location ID" in the server response as discussed on 
>> this list (but not yet in published draft), which the client could 
>> choose to associate with the client's internal records.
>>
>> The server periodically (server policy) issues a set of "potentially 
>> impacted" ids, where "potentially impacted" has already been 
>> discussed on this list. Each set includes: the ids that have been 
>> "potentially impacted" since the previous set was issued; the 
>> time-range covered; and a TTL hint on how long until the server is 
>> likely to issue the next list.  The server keeps the sets going back 
>> some interval defined by server policy.
>>
>> Client sends a "discovery" query to server, where server response is: 
>> the url(s) of the most recent and past set(s); and a TTL hint on how 
>> long until next list likely to be released.  Client then queries any 
>> of these urls, and server responds with the set that includes: the 
>> ids that have been "potentially impacted" since the previous set was 
>> issued; the time-range covered; and a TTL hint on how long until the 
>> server is likely to issue the next list.
>>
>> For clients that want to be highly proactive for changes, the client 
>> can store the "unique location id" from <findServiceResponse> and 
>> associate to the client's internal records.  The client then 
>> periodically queries the server for the set(s) of impacted IDs, and 
>> client gets a copy of each set. The client uses these sets of 
>> impacted ids from the servers list to check the clients internal 
>> records, and client can then revalidate those.
>>
>> No more client-provided uris entirely removing the possibility of 
>> abuse by malicious actors.
>>
>> The data in the issued set could be further extended to convey 
>> upcoming potential impacts to ids that have not yet been impacted.
>>
>>
>> /Jeff/
>>
>> From: Ecrit <[e-mail]> On Behalf Of Caron, Guy
>> Sent: Thursday, September 9, 2021 7:52 AM
>> To: Brian Rosen <[e-mail]>; Randall Gellens 
>> <[e-mail]>
>> Cc: ECRIT <[e-mail]>
>> Subject: Re: [Ecrit] planned-changes: two questions
>>
>> I’m aligned with Brian on the proposal to embed a reverse HTTP 
>> transaction within an existing one. I think this is asking for 
>> trouble.
>>
>> Guy
>>
>> De : Brian Rosen <[e-mail]>
>> Envoyé : 8 septembre 2021 19:32
>> À : Randall Gellens <[e-mail]>
>> Cc : Caron, Guy <[e-mail]>; ECRIT <[e-mail]>
>> Objet : [EXT]Re: [Ecrit] planned-changes: two questions
>>
>> Inline.  I think we need other opinions.  Certainly, we don’t have 
>> rough consensus.
>>
>>
>>
>> ________________________________________________________
>> On Sep 8, 2021, at 5:45 PM, Randall Gellens 
>> <[e-mail]> wrote:
>>
>> Why is a subsequent transaction better than a parallel one? The 
>> subsequent transaction model delays knowledge of the state.
>> If the client doesn't see an immediate POST, it doesn't know if 
>> there's been an error or not.
>> Yes it does.  If it doesn’t see an immediate POST, then there was a 
>> problem and it has to try again.
>>
>> If the client sees a POST and sends an ID, it doesn't know if the 
>> server received it and stored the URI.
>> Technically true, but vanishingly small issue.  You had a complete 
>> HTTPS transaction, but somehow the client isn’t sure the server got 
>> the ID.
>>
>> If the server tries a POST but gets an error, it may retry but 
>> meanwhile the client may retry the query, since it didn't get a POST.
>> That's always going to be something the server has to deal with: 
>> re-enrollment of the URI.  The client gets a normal response, and 
>> that tells it the URI was accepted.
>>
>> Should a client retain IDs if it hasn't seen a POST?
>> No, lack of POST means URI is not enrolled
>>
>> Also, requiring that the client send an ID seems like it may 
>> constrain client architectures since any part of the client system 
>> that receives a POST must know which IDs are pending.
>> I don’t agree.  This is a pretty normal “I send you a magic 
>> number and you have to return it to me” thing.  There are no 
>> “pending IDs”.  This is the first ID, so you need to return it, 
>> and only it, but it’s the only ID the client has at this point.
>>
>> What is the objection to a parallel POST?
>> I was taught that you don’t embed an HTTP transaction in another 
>> with the same partner because things go wrong with timeouts and 
>> logic.  It’s not a parallel POST, it’s an HTTPS transaction one 
>> way that encloses another HTTPS transaction the other way.  You are 
>> dependent on the inner one completing before the outer one times out 
>> or has some other issue.  I just think that’s dangerous.  I will 
>> admit that I was taught this quite a long time ago (before HTTPS for 
>> sure) and things could have changed, but I can’t recall an API that 
>> does that.  Also note that keep alive works exactly the same way as 
>> the initial POST.  In your proposal, those are different.  That’s 
>> not much, but it’s some code.  The “test” transaction tells you 
>> that the mechanism works and your URI is being retained, whether that 
>> happens the first time, or a year later.
>>
>> --Randall
>> On 8 Sep 2021, at 14:22, Brian Rosen wrote:
>> I don’t think there is any fragility in the URI setup.  For a new 
>> URI, the client requests the server to keep it in a findService.  
>> That transaction concludes by sending an ID.  The server immediately 
>> sends the test notification, to which the client responds with the 
>> ID.  At that point both sides know the URI was accepted and the 
>> client is valid.  That may be repeated periodically so both sides 
>> know the other is happy with the arrangements.  If the client 
>> doesn’t get the test notification, it knows the URI isn’t 
>> accepted.  If the server doesn’t see the ID, it’s knows that was 
>> a bogus request, or there is some other problem, and it shouldn’t 
>> save the URI.
>>
>> I proposed a null ID as the test transaction flag.  That would be a 
>> piece of XML with the right namespace, the right element name, but no 
>> value.  I think that is adequate and meets Guy’s minimal mechanism 
>> criteria.  The same XML is always sent.  It either has a set of IDs 
>> or it’s empty.
>>
>> Brian
>>
>>
>> On Sep 8, 2021, at 5:11 PM, Randall Gellens 
>> <[e-mail]> wrote:
>>
>> The proposed mechanism seems fragile. A client has no way to know if 
>> its request to store a URI was accepted. That makes it hard to debug. 
>> If there are errors of any kind when the server initially verifies a 
>> URI, the client has no idea. Even if the client repeats the 
>> transaction, the server will silently discard the URI. To me, that's 
>> asking for problems.
>> In my proposal, when the client requests to be notified and provides 
>> a URI, the server immediately does a parallel POST to that URI. If it 
>> gets a 202 Accepted response to the POST (or we can choose a 
>> different value), it stores the URI and returns the query result. If 
>> it gets a different response, it does not store the URI and returns 
>> the query result with a uriNotStored warning. If a client is trying 
>> to get the server to launch an attack on a third party entity, that 
>> entity will likely not support the URI in the first place, or will 
>> likely return an error response. If you want additional verification, 
>> have the server include the queried location in its test POST. That 
>> way, if the client maliciously sent a URI of a third party LIS, that 
>> LIS will know it does not have a query with that location 
>> outstanding.
>> Doing a parallel POST from the server to the client is one small 
>> transaction; this mechanism provides immediate feedback to both 
>> client and server. If there are DNS failures or network congestion or 
>> whatever, the server returns uriNotStored and the client can try 
>> again later.
>> As for multiple IDs in a single notification POST, having the client 
>> specify in the initial request the maximum it is prepared to accept 
>> seems reasonable. We can require that clients support a minimum 
>> number. If a server has more than the maximum, it sends them in 
>> multiple POSTs, with whatever separation in time it chooses. A server 
>> can include fewer IDs in a POST than the client's maximum.
>> We can have a 'test' notification value that is used both for the 
>> initial verification test and for periodic keep-alive tests.
>> --Randall
>> On 8 Sep 2021, at 12:17, Brian Rosen wrote:
>> Okay, I think the three of us are converging,  Here is a restatement 
>> of your description:
>> 1) In a validation query, a Client can request to be notified when 
>> the proffered LI should be revalidated, and provides a URI to send 
>> the notifications to;
>> 2) In the validation response, the Server provides an ID that the 
>> Client associates with the LI it just validated.   The server may 
>> silently ignore repeated requests to store a URI where the test in 4 
>> below fails.
>> 3) Immediately thereafter, if the URI is new to the Server, the 
>> Server sends a ‘test’ notification to the URI, with an empty ID.
>> 4) The recipient at the URI is expected to respond with the ID 
>> provided in step 2. If it does, the Server stores the URI for future 
>> notifications.  If it does not, the server ignores the request to 
>> store the URI.
>> 5) Some time after, the Server notifies the Client of an upcoming 
>> planned change by sending a notification to the successfully tested 
>> URI with the location ID;
>> 6) The client revalidates each LI in its database that matches the ID 
>> as of the date of the planned change. If no ID matches, it is a no-op 
>> at the Client. Revalidations may also result in no-op at the Client.
>> 7) LIs at the Client that are invalidated by the planned change are 
>> modified in its database to be valid (which probably mean another 
>> revalidation cycle) with an effective date set to <revalidateAsoF> 
>> value.
>> 8) The Server may send ’test’ notifications to the URI without 
>> any ID as a form of “keep-alive”.  Any ID provided by the Server 
>> to the client may be used as the response to the test transaction
>>
>>
>> There has been a discussion of sending more than one ID in a 
>> transaction.  I think that is a decent idea, but I worry about how 
>> big that could be.  Either we put a hard limit in the text or have 
>> something in the response to the test transaction that specifies a 
>> size limit for that client.
>>
>> Brian
>>
>>
>> On Sep 7, 2021, at 8:31 PM, Caron, Guy <[e-mail]> wrote:
>>
>> 1) In a validation query, a Client can request to be notified when 
>> the proffered LI should be revalidated, and provides a URI to send 
>> the notifications to;
>> 2) In the validation response, the Server provides an ID that the 
>> Client associates with the LI it just validated;
>> 3) Immediately thereafter, the Server sends a ‘test’ notification 
>> to the URI, without any ID;\
>> [br[For every new ID?  Or just once?  I wanted this to be a one time 
>> registration.
>> [GC] Just once per offered URI.
>>
>>
>> 4) The recipient at the URI is expected to respond with the ID 
>> provided in step 2. If it does, the Server stores the URI for future 
>> notifications. If it does not, the Server [let’s pick one: reject 
>> silently the URI and block the Client permanently/provides a 
>> ‘uriNotStored’ warning response to the URI {not compatible with 
>> the current proposal to test outside of LoST}/reject silently the URI 
>> and block the Client temporarily/other?];
>> [br]I don’t think an explicit failure is a problem.  The LoST 
>> server can limit retries if it needs to.
>> [GC] Ok for HTTP failures but what I’m talking about is when it 
>> fails to return the ID, like in your DoS example.
>>
>>
>>
>> 5) Some time after, the Server notifies the Client of an upcoming 
>> planned change by sending a notification to the successfully tested 
>> URI with the location ID;
>> 6) The client revalidates each LI in its database that matches the ID 
>> as of the date of the planned change. If no ID matches, it is a no-op 
>> at the Client. Revalidations may also result in no-op at the Client.
>> 7) LIs at the Client that are invalidated by the planned change are 
>> modified in its database to be valid (which probably mean another 
>> revalidation cycle) with an effective date set to <revalidateAsoF> 
>> value.
>> [br]I wanted periodic keep alives.  How would that work?
>> [GC] You mean at step 3? As I mentioned below, I was wondering about 
>> the necessity for periodic tests. If the group is convinced it is 
>> needed, the Server can simply redo step 3 and the Client can respond 
>> with one or many IDs it has from that Server.
>>
>>
>> NOTICE TO RECIPIENT: This email, including attachments, may contain 
>> information which is confidential, proprietary, attorney-client 
>> privileged and / or controlled under U.S. export laws and regulations 
>> and may be restricted from disclosure by applicable State and Federal 
>> law. Nothing in this email shall create any legal binding agreement 
>> between the parties unless expressly stated herein and provided by an 
>> authorized representative of Comtech Telecommunications Corp. or its 
>> subsidiaries. If you are not the intended recipient of this message, 
>> be advised that any dissemination, distribution, or use of the 
>> contents of this message is strictly prohibited. If you received this 
>> message in error, please notify us immediately by return email and 
>> permanently delete all copies of the original email and any attached 
>> documentation from any computer or other media.
>> _______________________________________________
>> Ecrit mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/ecrit 
>> <https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fecrit&data=04%7C01%7Cjeff.martin%40comtechtel.com%7Cb6354e9963b1469a1a1808d9747845af%7Ca9a26e696ae040c1bd801ca6cc677828%7C0%7C0%7C637668882191609050%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NDkyikK2aPrzaE826qPjFrrlsn1WtrgpqjKx3KBKnQQ%3D&reserved=0>


> _______________________________________________
> Ecrit mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ecrit