Re: Atom Link Extensions Use Case

[James M Snell <[e-mail]>]

Good stuff. I can definitely resurrect the draft and move it forward
if there's enough interest.

- James

On Fri, Jun 8, 2012 at 6:04 AM, Ed Summers <[e-mail]> wrote:
> Hi all,
>
> I am using Atom to syndicate access to data dumps at the Library of
> Congress. We have a web application that provides access to historic
> newspapers [1], and we have received requests for access to the
> underlying OCR data for research and commercial purposes. Despite the
> fact that this is historic data, we are routinely adding new content
> as it is digitized. Rather than require clients to issue millions of
> requests to get at the OCR data (which is actually web addressable)
> the plan is to periodically create a tarred and compressed dump file
> of new OCR content, and publish the availability of the file in an
> Atom feed, which interested parties can subscribe to. It's a similar
> model to what Wikimedia does for various Wikipedia projects [2].
>
> Here's a minimal example, to give you an idea of what I mean (warning
> URLs don't currently resolve):
>
> <?xml version="1.0" encoding="utf-8"?>
> <feed xmlns="http://www.w3.org/2005/Atom">
>    <title>Chronicling America OCR Dumps</title>
>    <link rel="self" type="application/atom+xml"
> href="http://chroniclingamerica.loc.gov/dumps/ocr/feed/" />
>    <id>info:lc/ndnp/dumps/ocr</id>
>    <author>
>        <name>Library of Congress</name>
>        <uri>http://loc.gov</uri>
>    </author>
>    <updated>2012-06-08T08:35:27-04:00</updated>
>    <entry>
>        <title>part-00001.tar.bz2</title>
>        <link rel="alternate" type="application/x-bzip2"
> href="http://chroniclingamerica.loc.gov/data/dumps/ocr/part-00001.tar.bz2"
> />
>        <id>info:lc/ndnp/dump/ocr/part-00001.tar.bz2</id>
>        <updated>2012-06-07T13:57:23-04:00</updated>
>        <summary type="xhtml"><div
> xmlns="http://www.w3.org/1999/xhtml">OCR dump file <a
> href="http://chroniclingamerica.loc.gov/data/dumps/ocr/part-00001.tar.bz2">part-00001.tar.bz2</a>
> with size 162.7 MB generated June 7, 2012, 1:57 p.m.</div></summary>
>    </entry>
> </feed>
>
> So the reason why I am writing here is that I would like to add
> checksum information to the feed to let clients verify that they have
> downloaded the data dump file correctly. An argument could be made
> that it's not necessary since a corrupted bz2 file would likely not
> decompress. An argument could also be made that the Content-MD5 header
> could be used. But I like the idea of making an explicit assertion
> about the checksum in the Atom document.
>
> After a bit of googling I ran across James Snell's Atom Link
> Extensions draft, which provides a pattern for including an md5
> checksum in the <link> element like so:
>
>    <link rel="alternate" type="application/x-bzip2"
> hash="md5:579758192095fde80896058af4ce0aee"
> href="http://chroniclingamerica.loc.gov/data/dumps/ocr/part-00001.tar.bz2"
> />
>
> Unfortunately it looks like the draft has expired. I was wondering:
>
> a) are there other established patterns for adding checksum
> information for resources in Atom
> b) if it's worth it for James to update the draft and try to push it
> forwards to an Informational status
>
> As more and more data providers make dumps of their data available to
> reduce crawling (like Wikipedia) it seems like a good use case for
> Atom to support.
>
> //Ed
>
> [1] http://chroniclingamerica.loc.gov
> [2] http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-abstract.xml-rss.xml
> [3] http://tools.ietf.org/html/draft-snell-atompub-link-extensions-08