Re: ietf-nntp Issue: number range rollover

[USENET news manager <[e-mail]>]

Clive D.W. Feather (hi to you, too, Clive!) wrote:
>> If Usenet (as an example ;-) continues to grow as it has, we may eat this
>> space pretty quickly in which case we either need to expand the range to
>> 64 bits and/or precisely define how to handle rollovers.
>
>There's no existing practice for handling such rollovers. Here's some
>possibilities:
>...
>(4) Make article numbers be modulo 2^32. Actually, this idea can be
>improved on. Divide the number space into 4 quadrants:
>    Q0 = 1         to 2^30-1
>    Q1 = 2^30      to 2^31-1
>    Q2 = 2^31      to 2^31+2^30-1
>    Q3 = 2^31+2^30 to 2^32-1
>Some time after the low water mark for a group enters Q3 and before the
>high water mark is reaching danger, the server should reset the group by
>subtracting 2^31 from all article numbers. If the client has a recorded HWM
>(or LWM) in Q3, and sees a GROUP response in Q1, it should assume this has
>happened and adjust its records in the same way. Obviously, the reset has
>to be atomic, but this could be done on Unix by:
>    - link each article to its new number
>    - change the active file
>    - wait for all clients using the old numbers to disconnect (or
>      forcibly disconnect them);
>    - remove the links with the old numbers.

"It's not that simple."

Thinking in terms of INN's current implementation:

(1) If anyone's reading news via NFS or direct local spool access on the
server (yes, it still happens!), the news server has no way of knowing 
when clients have stopped using the old numbers.

(2) The Xref: header uses the article numbers, and is stored in both the
article and the overview data. Some newsreaders use it to mark articles read
in all cross-posted groups, and the obvious client implementation (just update
.newsrc or equivalent) will not have the intended effect if any of the groups
other than where the article is being read have been reset. 

(3) The history file includes article filenames, and history entries may 
be retained for any period (determined by expiry etc.), possibly weeks or 
months. Retrieval by message ID may rely on the old article numbers until
expiry. 

(4) Outbound feeds rely on the article filenames, hence article numbers, and
a backlogged feed may rely on the article numbers remaining valid for days or
weeks. 

(5) A client may never have seen articles numbered in the Q3 range before
the reset occurs and it encounters articles in the Q1 range - or given a long
enough delay before the user re-visits the group and a high enough volume of 
articles, even the Q2 or Q3 range! 

More-or-less fixable, but not as simple as suggested by the summary quoted 
above! 

2 & 3 could be fixed up in the server - massaging the returned data - for
NNTP access, but would be a problem for direct/NFS access to the article,
overview, and history files. 4 would need all programs used for managing
outbound feeds (not just those supplied as standard with the server,
alternatives from other sources as well, e.g. nntplink, innfeed) to
understand the situation. Indeed, that's a variant of the problems arising
with newsreaders using direct or NFS access to the files. 

Removing the links to the high-numbered articles seems like an obvious
candidate for special-case handling during expiry, which could also update
the article numbers in the history entries (assuming an INN-like history
database, rewritten in full during expiry).

Some sort of "when this happens, the client should ..." solution to the 
rollover problem still seems the best option, in spite of these 
complications - it just needs the awkward cases sorted out cleanly!

Another complication: in configurations with INN slave servers or 
equivalent, explicitly using the same article numbers as a master server,
there'd need to be some mechanism to synchronise resetting the article 
numbers on slave servers.

And finally ... a random potential problem of a different sort - there might
not be sufficient free inodes to double up all the directory entries... and
that would be messier to handle than running out while accepting a new
article.

                                John Line
-- 
Cambridge University Computing Service - USENET news manager. Usually John Line
[e-mail]    (alias {newsmaster,news,usenet}@news.cam.ac.uk)