Re: [Tzdist] Fwd: [tzdist] #32 (service): managing historical data

[Lester Caine <[e-mail]>]

On 16/12/14 21:33, Cyrus Daboo wrote:
> Hi Doug,
> 
> --On December 16, 2014 at 1:20:15 PM -0700 Doug Royer
> <[e-mail]> wrote:
> 
>>> As I proposed before I want to see the version number be part of the
>>> meta-data associated with the time zone data - i.e., part of the list
>>> action response. I don't want to see it in the time zone data due to
>>> the fact that, with the current manner in which IANA tz data is
>>> "tagged" with a version, the entire set of time zones would be marked
>>> as changed.
>>
>> That point is valid, however as soon as the data is saved on the
>> tzdist-server, and the first fetch is performed, is it not just as likely
>> that it would trigger a everything is out of date when they compare it to
>> 'nothing authoritative known about this TZID yet - so get it' ?
> 
> OK, so when a list action changedsince is done we need to make sure of
> the following:
> 
> 1) The server only reports a time zone as changed if the underlying data
> or meta-data (other than version id) has changed.
The version is not part of the data. And if one is serving a 'modern'
truncated data set then and changes to TZ outside scope of the period
are ignored as well.

> 2) The server always reports the latest versions now in use for each
> publisher.
This is the bit that bothers me.
And it relates to your incremental changes, so there may be several
'versions' uncovered by a list/changedsince. So the server has to
produce a concatenated list based on all the versions between the old
and new version. Not a particular problem with the right caching store,
but another race hazard.

> So #1 allows clients to fetch the updated tzdata and update any
> meta-data associated with a "changed" time zone.
As indicated previously ...  one does not need to provide the whole TZid
period. Only the diff needs to be transmitted and this simplifies the
checking of material which may be using the updated period. Transmitting
the whole data set does then require the client to establish what the
change was and then assess the knock on effects. Discussion on package
size may be mute if processing a change set becomes the bottleneck?

> #2 allows the client to update the version id meta-data for any other
> time zone it has cached, but was not reported in #1.
While the implementation is a matter for the server software, I envision
a system which would have a database of TZid records which is read and
as a TZid record is updated it is added keyed by it's version number.
When one asks for a 'changedsince', the database returns all of the
versions up until latest. At any time one can read a version as a
complete set of TZid's or the diff's for a version number. I envisage
the same server working as the client cache so all that ever needs to be
transferred is the update diff. The problem arises where there are two
changes to a TZid in the window. Operationally one may only need the
second, but later the first one may be important for archival reasons.

> Note that, if a publisher chooses to include a version number in time
> zones they publish, and they ensure those only change when the data
> changes, then having the version embedded in the time zone data works
> fine as part of #1.
I think the problem people are having is that some caching processes do
treat the version as a variable. Code management systems where the
version number is in every file are a right pain when only the odd file
actually changed. As long as the version is only used as a tag to
identify a particular view then the caching system need only transfer
those 'files' which actually have changes. ETag can safely contain a
version since the list/changedsince mechanism will only list TZid's that
have changed. Users can still access any TZid at any version ...
bypassing the list/changedsince mechanism but if the data is unchanged
from the currently stored version then it can be thrown away.

I think it is probably worth pointing out that probably 90% of the
initial data can be installed as currently happens as part of the OS
installation. Then only updates from that base version need to be
supplied by tzdist. The installed version is a base version ID and
tzdist simply starts up with a list/changedsince from that base. But in
future that base install would also include the historic versions from
the start point that tzdist becomes active. The extra historic versions
are unlikely to be more than a percentage of the total data.

-- 
Lester Caine - G8HFL
-----------------------------
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk
Rainbow Digital Media - http://rainbowdigitalmedia.co.uk