Re: [Ecrit] planned-changes: two questions

[Brian Rosen <[e-mail]>]

Minutes?

So the clients of this service poll for changes every few minutes?

What is a “partial civic address”?  Example?  Something like an A1/A2/A4 but no street name/number, meaning any address in this unincorporated community name?

ISTM that a planned change quite often can’t be fixed in a way that code could do it.  So, we still need to advertise the change well in advance of the change, and have the ability to do the AsOf validation.  Changing the way the client discovers that a change is coming is one thing, not knowing well in advance is, I think, unacceptable. 

Brian




> On Sep 10, 2021, at 12:13 PM, Dan Banks <dbanks=[e-mail]> wrote:
> 
> I am opposed to asking LoST servers to store URIs as well.  I believe it significantly overcomplicates the solution and is the biggest reason why I have not supported the planned changes draft.
>  
> Several years ago we implemented a mechanism to allow a LIS to know that it needs to revalidate.  Roughly described, that mechanism consists of a web API that can be queried for changes after a particular time.  The response is a summary of changes in terms of partial civic addresses that allow the LIS to identify records that are likely to be affected, or possibly that there have been so many changes that a full revalidation is recommended.  We’ve made the API discoverable via U-NAPTR using the same app string as a given LoST server.
>  
> Yes, the LIS has to poll the various LoST servers and ask if there have been any changes.  That is a cheap operation to implement, and the practical difference between knowing the very instant a change is published versus finding out within a few minutes is not significant in any way.
>  
> Dan
>  
> From: Ecrit <[e-mail]> On Behalf Of Jeff Martin
> Sent: Thursday, September 9, 2021 4:09 PM
> To: [e-mail]
> Subject: Re: [Ecrit] planned-changes: two questions
>  
> Asking a server to store and later user a client-provided uri opens the door for abuse by malicious actors.   Which got me to thinking, what if we removed the client-provided uri entirely?  If no uris then no risk of abuse.
>  
> Keep the new optional <plannedChange asOf="..."> but without 'uri' in the client's <findServiceRequest>.  Keep the new optional <ttl> in the server's <findServiceResponse> as a hint to clients that may want to initiate revalidation from the client side.  Keep the new optional server "unique location ID" in the server response as discussed on this list (but not yet in published draft), which the client could choose to associate with the client's internal records.  
>  
> The server periodically (server policy) issues a set of "potentially impacted" ids, where "potentially impacted" has already been discussed on this list. Each set includes: the ids that have been "potentially impacted" since the previous set was issued; the time-range covered; and a TTL hint on how long until the server is likely to issue the next list.  The server keeps the sets going back some interval defined by server policy.
>  
> Client sends a "discovery" query to server, where server response is: the url(s) of the most recent and past set(s); and a TTL hint on how long until next list likely to be released.  Client then queries any of these urls, and server responds with the set that includes: the ids that have been "potentially impacted" since the previous set was issued; the time-range covered; and a TTL hint on how long until the server is likely to issue the next list.
> 
> For clients that want to be highly proactive for changes, the client can store the "unique location id" from <findServiceResponse> and associate to the client's internal records.  The client then periodically queries the server for the set(s) of impacted IDs, and client gets a copy of each set. The client uses these sets of impacted ids from the servers list to check the clients internal records, and client can then revalidate those.
>  
> No more client-provided uris entirely removing the possibility of abuse by malicious actors. 
>  
> The data in the issued set could be further extended to convey upcoming potential impacts to ids that have not yet been impacted.
>  
>  
> /Jeff/
>  
> From: Ecrit <[e-mail] <mailto:[e-mail]>> On Behalf Of Caron, Guy
> Sent: Thursday, September 9, 2021 7:52 AM
> To: Brian Rosen <[e-mail] <mailto:[e-mail]>>; Randall Gellens <[e-mail] <mailto:[e-mail]>>
> Cc: ECRIT <[e-mail] <mailto:[e-mail]>>
> Subject: Re: [Ecrit] planned-changes: two questions
>  
> I’m aligned with Brian on the proposal to embed a reverse HTTP transaction within an existing one. I think this is asking for trouble.
>  
> Guy
>  
> De : Brian Rosen <[e-mail] <mailto:[e-mail]>> 
> Envoyé : 8 septembre 2021 19:32
> À : Randall Gellens <[e-mail] <mailto:[e-mail]>>
> Cc : Caron, Guy <[e-mail] <mailto:[e-mail]>>; ECRIT <[e-mail] <mailto:[e-mail]>>
> Objet : [EXT]Re: [Ecrit] planned-changes: two questions
>  
> Inline.  I think we need other opinions.  Certainly, we don’t have rough consensus.
>  
>  
> 
> ________________________________________________________
> On Sep 8, 2021, at 5:45 PM, Randall Gellens <[e-mail] <mailto:[e-mail]>> wrote:
>  
> Why is a subsequent transaction better than a parallel one? The subsequent transaction model delays knowledge of the state.
> If the client doesn't see an immediate POST, it doesn't know if there's been an error or not.
> Yes it does.  If it doesn’t see an immediate POST, then there was a problem and it has to try again.
> 
> If the client sees a POST and sends an ID, it doesn't know if the server received it and stored the URI.
> Technically true, but vanishingly small issue.  You had a complete HTTPS transaction, but somehow the client isn’t sure the server got the ID. 
> 
> If the server tries a POST but gets an error, it may retry but meanwhile the client may retry the query, since it didn't get a POST.
> That's always going to be something the server has to deal with: re-enrollment of the URI.  The client gets a normal response, and that tells it the URI was accepted.
> 
> Should a client retain IDs if it hasn't seen a POST?
> No, lack of POST means URI is not enrolled
> 
> Also, requiring that the client send an ID seems like it may constrain client architectures since any part of the client system that receives a POST must know which IDs are pending.
> I don’t agree.  This is a pretty normal “I send you a magic number and you have to return it to me” thing.  There are no “pending IDs”.  This is the first ID, so you need to return it, and only it, but it’s the only ID the client has at this point.
> 
> What is the objection to a parallel POST?
> I was taught that you don’t embed an HTTP transaction in another with the same partner because things go wrong with timeouts and logic.  It’s not a parallel POST, it’s an HTTPS transaction one way that encloses another HTTPS transaction the other way.  You are dependent on the inner one completing before the outer one times out or has some other issue.  I just think that’s dangerous.  I will admit that I was taught this quite a long time ago (before HTTPS for sure) and things could have changed, but I can’t recall an API that does that.  Also note that keep alive works exactly the same way as the initial POST.  In your proposal, those are different.  That’s not much, but it’s some code.  The “test” transaction tells you that the mechanism works and your URI is being retained, whether that happens the first time, or a year later.
> 
> --Randall
> On 8 Sep 2021, at 14:22, Brian Rosen wrote:
> I don’t think there is any fragility in the URI setup.  For a new URI, the client requests the server to keep it in a findService.  That transaction concludes by sending an ID.  The server immediately sends the test notification, to which the client responds with the ID.  At that point both sides know the URI was accepted and the client is valid.  That may be repeated periodically so both sides know the other is happy with the arrangements.  If the client doesn’t get the test notification, it knows the URI isn’t accepted.  If the server doesn’t see the ID, it’s knows that was a bogus request, or there is some other problem, and it shouldn’t save the URI.
>  
> I proposed a null ID as the test transaction flag.  That would be a piece of XML with the right namespace, the right element name, but no value.  I think that is adequate and meets Guy’s minimal mechanism criteria.  The same XML is always sent.  It either has a set of IDs or it’s empty.
>  
> Brian
>  
> 
> On Sep 8, 2021, at 5:11 PM, Randall Gellens <[e-mail] <mailto:[e-mail]>> wrote:
>  
> The proposed mechanism seems fragile. A client has no way to know if its request to store a URI was accepted. That makes it hard to debug. If there are errors of any kind when the server initially verifies a URI, the client has no idea. Even if the client repeats the transaction, the server will silently discard the URI. To me, that's asking for problems.
> In my proposal, when the client requests to be notified and provides a URI, the server immediately does a parallel POST to that URI. If it gets a 202 Accepted response to the POST (or we can choose a different value), it stores the URI and returns the query result. If it gets a different response, it does not store the URI and returns the query result with a uriNotStored warning. If a client is trying to get the server to launch an attack on a third party entity, that entity will likely not support the URI in the first place, or will likely return an error response. If you want additional verification, have the server include the queried location in its test POST. That way, if the client maliciously sent a URI of a third party LIS, that LIS will know it does not have a query with that location outstanding.
> Doing a parallel POST from the server to the client is one small transaction; this mechanism provides immediate feedback to both client and server. If there are DNS failures or network congestion or whatever, the server returns uriNotStored and the client can try again later.
> As for multiple IDs in a single notification POST, having the client specify in the initial request the maximum it is prepared to accept seems reasonable. We can require that clients support a minimum number. If a server has more than the maximum, it sends them in multiple POSTs, with whatever separation in time it chooses. A server can include fewer IDs in a POST than the client's maximum.
> We can have a 'test' notification value that is used both for the initial verification test and for periodic keep-alive tests.
> --Randall
> On 8 Sep 2021, at 12:17, Brian Rosen wrote:
> Okay, I think the three of us are converging,  Here is a restatement of your description:
> 1) In a validation query, a Client can request to be notified when the proffered LI should be revalidated, and provides a URI to send the notifications to;
> 2) In the validation response, the Server provides an ID that the Client associates with the LI it just validated.   The server may silently ignore repeated requests to store a URI where the test in 4 below fails.
> 3) Immediately thereafter, if the URI is new to the Server, the Server sends a ‘test’ notification to the URI, with an empty ID.
> 4) The recipient at the URI is expected to respond with the ID provided in step 2. If it does, the Server stores the URI for future notifications.  If it does not, the server ignores the request to store the URI. 
> 5) Some time after, the Server notifies the Client of an upcoming planned change by sending a notification to the successfully tested URI with the location ID;
> 6) The client revalidates each LI in its database that matches the ID as of the date of the planned change. If no ID matches, it is a no-op at the Client. Revalidations may also result in no-op at the Client.
> 7) LIs at the Client that are invalidated by the planned change are modified in its database to be valid (which probably mean another revalidation cycle) with an effective date set to <revalidateAsoF> value.
> 8) The Server may send ’test’ notifications to the URI without any ID as a form of “keep-alive”.  Any ID provided by the Server to the client may be used as the response to the test transaction
>  
>  
> There has been a discussion of sending more than one ID in a transaction.  I think that is a decent idea, but I worry about how big that could be.  Either we put a hard limit in the text or have something in the response to the test transaction that specifies a size limit for that client.
>  
> Brian
>  
> 
> On Sep 7, 2021, at 8:31 PM, Caron, Guy <[e-mail] <mailto:[e-mail]>> wrote:
>  
> 1) In a validation query, a Client can request to be notified when the proffered LI should be revalidated, and provides a URI to send the notifications to;
> 2) In the validation response, the Server provides an ID that the Client associates with the LI it just validated;
> 3) Immediately thereafter, the Server sends a ‘test’ notification to the URI, without any ID;\
> [br[For every new ID?  Or just once?  I wanted this to be a one time registration.
> [GC] Just once per offered URI.
>  
> 
> 4) The recipient at the URI is expected to respond with the ID provided in step 2. If it does, the Server stores the URI for future notifications. If it does not, the Server [let’s pick one: reject silently the URI and block the Client permanently/provides a ‘uriNotStored’ warning response to the URI {not compatible with the current proposal to test outside of LoST}/reject silently the URI and block the Client temporarily/other?];
> [br]I don’t think an explicit failure is a problem.  The LoST server can limit retries if it needs to.
> [GC] Ok for HTTP failures but what I’m talking about is when it fails to return the ID, like in your DoS example.
>  
>  
> 
> 5) Some time after, the Server notifies the Client of an upcoming planned change by sending a notification to the successfully tested URI with the location ID;
> 6) The client revalidates each LI in its database that matches the ID as of the date of the planned change. If no ID matches, it is a no-op at the Client. Revalidations may also result in no-op at the Client.
> 7) LIs at the Client that are invalidated by the planned change are modified in its database to be valid (which probably mean another revalidation cycle) with an effective date set to <revalidateAsoF> value.
> [br]I wanted periodic keep alives.  How would that work?
> [GC] You mean at step 3? As I mentioned below, I was wondering about the necessity for periodic tests. If the group is convinced it is needed, the Server can simply redo step 3 and the Client can respond with one or many IDs it has from that Server. 
>  
>  
> NOTICE TO RECIPIENT: This email, including attachments, may contain information which is confidential, proprietary, attorney-client privileged and / or controlled under U.S. export laws and regulations and may be restricted from disclosure by applicable State and Federal law. Nothing in this email shall create any legal binding agreement between the parties unless expressly stated herein and provided by an authorized representative of Comtech Telecommunications Corp. or its subsidiaries. If you are not the intended recipient of this message, be advised that any dissemination, distribution, or use of the contents of this message is strictly prohibited. If you received this message in error, please notify us immediately by return email and permanently delete all copies of the original email and any attached documentation from any computer or other media.
> _______________________________________________
> Ecrit mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ecrit