Re: [Tzdist] Fwd: [tzdist] #32 (service): managing historical data

[Cyrus Daboo <[e-mail]>]

Hi Paul,

--On December 14, 2014 at 12:12:50 PM -0800 Paul Eggert 
<[e-mail]> wrote:

>> In that kind of environment, yes, every byte does count, because the sum
>> of all those extra bytes across the full set of users/devices is huge.
>
> In that case I'm puzzled why VTIMEZONE was chosen as the format for data,
> as it's pretty fat.  For America/Los_Angeles, vzic generates a VTIMEZONE
> format containing 4004 bytes, compared to the 1058 bytes that Android
> uses for tz binary format.  If the problem is that VTIMEZONE is bloated,
> perhaps a better solution would be to recommend a more-compact data
> format.

VTIMEZONE is used because the first consumers of this protocol are 
iCalendar-based products (and as has been stated before it was those 
products that started this whole thing off by first implementing their own 
tzdist-like behavior in a proprietary manner, with the subsequent 
realization that an interoperable protocol would be better for all). Yes it 
could be considered bloated compared to some binary representation, however 
it is an interoperable standard format that exists today.

> That being said, I'm still puzzled as to why shrinking the byte count is
> so important that we should complicate the protocol to support it.

I think the protocol as it stands is far from complicated. It relies on 
ETags (which are a fundamental element of HTTP that nearly all devices 
support in their HTTP stacks), and the dtstamp/changedsince behavior, that 
I am sure all the other implementors of the current protocol will claim, 
like me, is trivial to implement and provides an incremental update 
mechanism that is more efficient at very little added cost.

The complexity is all being added by the apparent need for versioning. I 
still contend that is unnecessary for tzdist. It is a problem for the 
consumers of the tzdist data and tzdist itself provides sufficient 
meta-data for them to deal with that.

> As a
> back-of-the-envelope calculation, suppose a VTIMEZONE zone is 4000 bytes
> and we make the same assumptions as before for everything else (32-bit
> native tz binary data is 1100 bytes, the tzdist request is 200 bytes, the
> protocol overhead is 600 bytes for both request and response, and there
> are 10 updates per year). Then we see the following per-client costs over
> time:
>
> 0.0137 b/s VTIMEZONE full copy each request
> 0.0063 b/s 32-bit tz binary full copy each request
> 0.0035 b/s tzdist version-style no-change handshake
> 0.0030 b/s? ETag-style no-change handshake? (not clear what's involved
> here)
>
> To give us some perspective on this, if we assume a single
> reasonably-cheap server can process 300 Mb/s sustained, then a single
> tzdist server could support 21 billion tzdist clients each using the
> least-efficient protocol mentioned above.  That's pretty scalable: one
> cheapish server should be able to handle the entire planet and then some.
> So why are we worrying so much about making every byte count?  Shouldn't
> it be more important to keep things simple?

Scaleability involves a lot more than just the server by itself, there are 
many other issues related to intermediaries, to efficiency on the client 
side (where battery drain on devices can be a show stopper), security 
requirements, etc. The systems I have built have not had to scale much 
beyond 100K's of users, but I work with others who do build out to 10's 
millions or more. Simple back of the envelope calculations don't suffice - 
there are far too many variables involved - but the efficiency of the base 
protocol itself is one major aspect that we do have control over when we 
design it. Even in the systems I have built, protocol efficiency has been 
one of the biggest pieces to deal with in terms of scaleability.

I think, as described earlier, what we have right now is a pretty lean 
protocol that provides a reasonable path to scaleable implementations, and 
we have extensibility to go further if what we have done to date is not 
sufficient for the very large providers.


-- 
Cyrus Daboo