Re: [Ecrit] planned-changes: two questions

[Brian Rosen <[e-mail]>]

Yes, the server can control the rate of revalidation by spacing out notifications, but so far the client can’t control the size of one notification.  If I built the LVF on any modern self scaling infrastructure, it could easily handle a million revalidations, one at a time in a short time by a bunch of clients, but a single client may fall over with a single 256M byte transaction.

I say this watching a web server do something weird on a 4Mbyte transaction.

I’m okay with a client getting 1M or 10M IDs notified in a short period of time, but not in a single transaction.

Brian

> On Sep 8, 2021, at 4:03 PM, Caron, Guy <[e-mail]> wrote:
> 
> Ah, your concern was for the Client. But I think the Server has more at stake here. If it puts too much IDs in the notification, it will be bombarded by revalidation requests that it may not be able to handle. It your large amalgamation example, if I was the Server, I would spread the notifications over a number of days (we probably should state that the Client SHOULD perform the revalidations within x hours of the notification). The Client would have to live with a large number of IDs anyway.
>  
> Guy
>  
> De : Brian Rosen <[e-mail]> 
> Envoyé : 8 septembre 2021 15:54
> À : Caron, Guy <[e-mail]>
> Cc : Randall Gellens <[e-mail]>rg>; ECRIT <[e-mail]>
> Objet : [EXT]Re: [Ecrit] planned-changes: two questions
>  
>  
> 
> 
> On Sep 8, 2021, at 3:43 PM, Caron, Guy <[e-mail] <mailto:[e-mail]>> wrote:
>  
> I think we are.
>  
> Small nit in step 7: <revalidateAsoF> should be <revalidateAsOf>
> Yes
> 
>  
> Related questions:
> 1.       Should it be exactly on <revalidateAsOf>?
> 2.       What will be the precision of that datetimestamp, hour, minute, second, sub-second (type="xs:dateTime" supports that level of granularity)?
> I get the issue.  I think we should be reasonable.  Let’s say it’s a minute of granularity, and that a request to validate asOf the dateTime of revalidayeAsOf should be using the new LI.  So revalidateAsOf is the time new data is present, not the last time old data was present.
> 
>  
> To your question about multiple IDs in the notification (I assume this is what you meant by “more than one ID in a transaction”), I would leave it to the Server to determine. The Client should abide to the list of IDs it gets notification for but nothing precludes a Client to perform validations outside of the planned-changes mechanism. In fact, a Client may choose to revalidate its entire database periodically even if it supports planned-changes. I don’t like it but there is no text that precludes this to happen.
> What I’m worried about is a million IDs in a single transaction when a city annexes a large suburb.  The client has to be prepared to accept the largest transaction the server will issue.  I want to either put a hard limit on it, or allow the client to tell the server how big the transaction is.  This is in units of IDs.  We limited the size of URIs, but with one URI per client, we can get rid of that.  If we allowed an ID to be as big as 256 bytes, a million IDs is 256Mbytes.  I don’t think clients should be obligated to accept that in one POST. 
>  
> 
> 
>  
> Guy
>  
> De : Brian Rosen <[e-mail] <mailto:[e-mail]>> 
> Envoyé : 8 septembre 2021 15:18
> À : Caron, Guy <[e-mail] <mailto:[e-mail]>>
> Cc : Randall Gellens <[e-mail] <mailto:[e-mail]>>; ECRIT <[e-mail] <mailto:[e-mail]>>
> Objet : [EXT]Re: [Ecrit] planned-changes: two questions
>  
> Okay, I think the three of us are converging,  Here is a restatement of your description:
> 1) In a validation query, a Client can request to be notified when the proffered LI should be revalidated, and provides a URI to send the notifications to;
> 2) In the validation response, the Server provides an ID that the Client associates with the LI it just validated.   The server may silently ignore repeated requests to store a URI where the test in 4 below fails.
> 3) Immediately thereafter, if the URI is new to the Server, the Server sends a ‘test’ notification to the URI, with an empty ID.
> 4) The recipient at the URI is expected to respond with the ID provided in step 2. If it does, the Server stores the URI for future notifications.  If it does not, the server ignores the request to store the URI. 
> 5) Some time after, the Server notifies the Client of an upcoming planned change by sending a notification to the successfully tested URI with the location ID;
> 6) The client revalidates each LI in its database that matches the ID as of the date of the planned change. If no ID matches, it is a no-op at the Client. Revalidations may also result in no-op at the Client.
> 7) LIs at the Client that are invalidated by the planned change are modified in its database to be valid (which probably mean another revalidation cycle) with an effective date set to <revalidateAsoF> value.
> 8) The Server may send ’test’ notifications to the URI without any ID as a form of “keep-alive”.  Any ID provided by the Server to the client may be used as the response to the test transaction
>  
>  
> There has been a discussion of sending more than one ID in a transaction.  I think that is a decent idea, but I worry about how big that could be.  Either we put a hard limit in the text or have something in the response to the test transaction that specifies a size limit for that client.
>  
> Brian
> 
> 
> 
> On Sep 7, 2021, at 8:31 PM, Caron, Guy <[e-mail] <mailto:[e-mail]>> wrote:
>  
> 1) In a validation query, a Client can request to be notified when the proffered LI should be revalidated, and provides a URI to send the notifications to;
> 2) In the validation response, the Server provides an ID that the Client associates with the LI it just validated;
> 3) Immediately thereafter, the Server sends a ‘test’ notification to the URI, without any ID;\
> [br[For every new ID?  Or just once?  I wanted this to be a one time registration.
> [GC] Just once per offered URI.
> 
> 
> 
> 
> 4) The recipient at the URI is expected to respond with the ID provided in step 2. If it does, the Server stores the URI for future notifications. If it does not, the Server [let’s pick one: reject silently the URI and block the Client permanently/provides a ‘uriNotStored’ warning response to the URI {not compatible with the current proposal to test outside of LoST}/reject silently the URI and block the Client temporarily/other?];
> [br]I don’t think an explicit failure is a problem.  The LoST server can limit retries if it needs to.
> [GC] Ok for HTTP failures but what I’m talking about is when it fails to return the ID, like in your DoS example.
>  
> 
> 
> 
> 
> 5) Some time after, the Server notifies the Client of an upcoming planned change by sending a notification to the successfully tested URI with the location ID;
> 6) The client revalidates each LI in its database that matches the ID as of the date of the planned change. If no ID matches, it is a no-op at the Client. Revalidations may also result in no-op at the Client.
> 7) LIs at the Client that are invalidated by the planned change are modified in its database to be valid (which probably mean another revalidation cycle) with an effective date set to <revalidateAsoF> value.
> [br]I wanted periodic keep alives.  How would that work?
> [GC] You mean at step 3? As I mentioned below, I was wondering about the necessity for periodic tests. If the group is convinced it is needed, the Server can simply redo step 3 and the Client can respond with one or many IDs it has from that Server. 
>  
> External Email: Please use caution when opening links and attachments / Courriel externe: Soyez prudent avec les liens et documents joints
>  
> External Email: Please use caution when opening links and attachments / Courriel externe: Soyez prudent avec les liens et documents joints