Re: [ftpext] I-D Action:draft-ietf-ftpext2-hash-00.txt

Anthony Bryan <anthonybryan@gmail.com> Fri, 26 November 2010 08:46 UTC

Return-Path: <anthonybryan@gmail.com>
X-Original-To: ftpext@core3.amsl.com
Delivered-To: ftpext@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 54D9B3A6A63 for <ftpext@core3.amsl.com>; Fri, 26 Nov 2010 00:46:05 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.577
X-Spam-Level:
X-Spam-Status: No, score=-2.577 tagged_above=-999 required=5 tests=[AWL=-0.278, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cOGqRuX3sEeU for <ftpext@core3.amsl.com>; Fri, 26 Nov 2010 00:46:04 -0800 (PST)
Received: from mail-ey0-f172.google.com (mail-ey0-f172.google.com [209.85.215.172]) by core3.amsl.com (Postfix) with ESMTP id 0A7463A69A6 for <ftpext@ietf.org>; Fri, 26 Nov 2010 00:46:03 -0800 (PST)
Received: by eyd10 with SMTP id 10so875634eyd.31 for <ftpext@ietf.org>; Fri, 26 Nov 2010 00:47:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=R+oy6pMVzqKcEKnRkpICdgJwS+lmwOtwv5DU+qEUeO0=; b=Yp+cAW8beFisQaINWdrnO/qCBJkXSzCJ8crYo1p1WF5laZOb/JOBC+1CyA6+vV4Cew fboVBqFXCebGBI5ndY0qtWKkUvqQ2Xc8aJq9dhpWwCYUIggx/A9BF9T0iSVULfFIW3hZ UPr1rxFyi7/h96dO1P2yZhhly1uohlPm9GK2o=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=DGxlZf8JEU3Y/wKOhJXZ6wU3HTZyu9x8o+F5p9xXF73PLHLzIDVk+XGjmM0+P0vw/r ecHZ95C4Zu3oe5G1cMxj7GIRQnqw3fMNxqzUyqpOeU7FJomQ087mYHMOn16ZYEcaXSBl ZlDO17ByurpH4fFsqBoPl+FpqM9ph9Ctg+LK4=
MIME-Version: 1.0
Received: by 10.213.22.197 with SMTP id o5mr6065859ebb.89.1290761226519; Fri, 26 Nov 2010 00:47:06 -0800 (PST)
Received: by 10.213.28.2 with HTTP; Fri, 26 Nov 2010 00:47:06 -0800 (PST)
In-Reply-To: <4CEF00C0.5050200@gmail.com>
References: <20101124224501.31531.96663.idtracker@localhost> <4CEDAA9A.8030303@kimmeringer.de> <AANLkTik8Bv9YKo+uPt0ntk9OS0qQW2T5_RJ56CFZxMnS@mail.gmail.com> <4CEF00C0.5050200@gmail.com>
Date: Fri, 26 Nov 2010 03:47:06 -0500
Message-ID: <AANLkTimFeub1ynp4UcKAv84GP9f=gcG485jUgT82mB87@mail.gmail.com>
From: Anthony Bryan <anthonybryan@gmail.com>
To: =?ISO-8859-1?Q?=C1ngel_Gonz=E1lez?= <keisial@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: ftpext@ietf.org
Subject: Re: [ftpext] I-D Action:draft-ietf-ftpext2-hash-00.txt
X-BeenThere: ftpext@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <ftpext.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/ftpext>, <mailto:ftpext-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ftpext>
List-Post: <mailto:ftpext@ietf.org>
List-Help: <mailto:ftpext-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ftpext>, <mailto:ftpext-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 26 Nov 2010 08:46:05 -0000

2010/11/25 Ángel González <keisial@gmail.com>:
> Anthony Bryan wrote:
>>> I read the draft and tried to find the reason by searching the
>>> archive. Why are startpoint and endpoint (SP and EP) missing?
>> good question!
>>
>> as you say, some of the other commands (XCRC, XMD5, XSHA1) support
>> partial file hashing.
>>
>> I remember this was on our list of known issues, and we authors
>> brought it up a number of times. we never received any feedback on it
>> (except for Mat from SmartFTP on his forum who was for it), so it was
>> dropped.
>>
>> [for everyone's info, some of the older FTP drafts like HASH that were
>> around before this mailing list were discussed on apps-discuss (going
>> back to March for HASH):
>> http://www.ietf.org/mail-archive/web/apps-discuss/current/maillist.html
>> ]
>
> I also think it is interesting to have hashes of partial content.
> They key piece is 'partial verification'. You could have distributed
> downloads
> from several mirrors without being able to separatedly hash each piece.
> Hash the file on all the mirrors (so it's the same), download the pieces,
> join, hash the final file. If it's good, ok. If not, redownload
> *everything* again.
> Which is exactly what you would have to do if you did a normal download from
> the ftp.
> You download a big file (eg. a DVD iso) and it fails verification (the
> server hash
> command is analogous to the checksums usually provided in the dowload page).
> That file is quite worthless now, even if you know that the error will
> probably be
> just a couple of bytes.
> Let's assume now that the ftp server allos a hash range command. Then,
> instead
> of redownloading everything, your client starts bisecting the file,
> finds a 1K block
> where your local copy differs, and redownloading it you're done.

yep, the partial file hashes are for download repair.

that's what we use them for in Metalink, but instead of being able to
request the hash for a specific range, the hashes are pre-computed &
stored in XML for (usually 256k but can be whatever size) chunks,
instead of dynamically requested.

as mentioned, Content-MD5 (RFC 1864) requests partial hashes for HTTP,
and Instance Digests (RFC 3230) for full file hashes.

> A couple of requirements:
> * The user should not be required to know the file size in advance.

you mean the client? why? it should just assume from a starting point
to the end of the file?

> * The protocol should allow a server allowing full file hashes but
> rejecting partial hashes.

right, partial file hashing would be OPTIONAL.
-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads