Re: [ftpext] COMB command IETF draft proposal

[Van Glass <[e-mail]>]

I think that having the server perform the delete tasks is preferred
for the simple reason that this would reduce the number of commands
going across the command channel.  We have not yet implemented COMB
but would like to pursue it.  With that said I'm curious to know your
experience COMBing a large number of files or very large files.  Have
you experienced any issues where the COMBine process on the server
side takes a long amount of time, possibly resulting in a command
channel timeout on the client side?

Thanks,

Van Glass
JSCAPE
Managed File Transfer and Network Solutions
http://www.jscape.com




On Wed, Jun 8, 2011 at 1:59 PM, Robert Oslin <[e-mail]> wrote:
> Thanks for the feedback! Issuing a hash (or XCRC until HASH is formally ratified) makes a lot of sense and will be inserted as a strongly recommended practice (i.e. SHOULD). As for the delete operation, I would be careful to make it mandatory of the client, rather leaving it open to interpretation, as there are many ways for the server to execute the combine operation, which may or may not result in the original parts being available for deletion. For example a rudimentary concatenate operation by the server on ascii type files (which would be quite slow) would result in each part being removed by nature of the concatenation operation, hence a subsequent delete operation would result in 5xx errors returned to the client on each DELE request. Perhaps it would be wise to recommend that the client check for parts post operation and do cleanup if any parts remain? In our particular implementation (and in the handful of servers that support COMB so far), it is the server that does
>  the cleanup as a function of the combine operation. Let me know if that makes sense, or if not, then how to deal with the myriad ways in which servers may go about combining files and how to deal with corner cases.
>
> Robert Oslin
>
> -----Original Message-----
> From: Robert McMurray [mailto:[e-mail]]
> Sent: Wednesday, June 08, 2011 2:11 PM
> To: Robert Oslin; [e-mail]
> Subject: RE: [ftpext] COMB command IETF draft proposal
>
> Thanks, Robert.
>
> I like the idea behind the COMB command; I could see where this could be quite useful for transferring large files.
>
> One consideration that I have in your draft is verbiage that "The server-FTP process SHOULD delete each of the parts identified by the COMB command once the target file is created." I would think that the cleanup behavior for partial files would be left to the client, since the remaining parts of the draft place the responsibility on the client for splitting the file and keeping track of the parts. For example:
>
>      C> STOR foo.txt
>      S> 226 Transfer complete.
>      C> STOR bar.txt
>      S> 226 Transfer complete.
>      C> COMB foobar.txt foo.txt bar.txt
>      S> 250 COMB command successful.
>      C> DELE foo.txt
>      S> 250 DELE command successful.
>      C> DELE bar.txt
>      S> 250 DELE command successful.
>
> As you mentioned in your draft, it would be advantageous if COMB was used with the HASH command to verify data integrity before the client takes care of the deletion tasks. For example:
>
>      C> OPTS HASH MD5
>      S> 200 MD5
>      C> STOR foo.txt
>      S> 226 Transfer complete.
>      C> STOR bar.txt
>      S> 226 Transfer complete.
>      C> COMB foobar.txt foo.txt bar.txt
>      S> 250 COMB command successful.
>      C> HASH foobar.txt
>      S> 213 MD5 426f62526f636b73 foobar.txt
>      C> DELE foo.txt
>      S> 250 DELE command successful.
>      C> DELE bar.txt
>      S> 250 DELE command successful.
>
> Thanks again.
>
> Robert McMurray
> [e-mail]
>
> ------------------------------
> From: Robert Oslin [mailto:[e-mail]]
> Sent: Wednesday, June 08, 2011 9:15 AM
> To: [e-mail]
> Subject: [ftpext] COMB command IETF draft proposal
>
> "Any submission to the IETF intended by the Contributor for publication as all or part of an IETF Internet-Draft or RFC and any statement made within the context of an IETF activity is considered an "IETF Contribution".
>
> It only took me 10+ years, but here's my contribution the FTP ext community (attached).
>
> Essentially an early draft to ratify the commonly used COMB command for support of multi-part (a.k.a accelerated) uploads. I welcome comments/questions from the community. Ideally redlining the document (or via comments in the doc). I will resubmit using formal IETF draft (in ASCII) once I've addressed all comments/questions in the MS Word version (hope that's ok and doesn't break and IETF rules).
>
> Thanks,
>
> Robert Oslin
> Director of Product Management
> Tel: [phone]
> Fax: [phone]
> Send me large files securely
>
> www.globalscape.com
>    (NYSE Amex:GSB) *This communication, including attachments, is for the exclusive use of the addressee and may contain proprietary, confidential or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this communication and destroy all copies.
>
>
> _______________________________________________
> ftpext mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ftpext
>