Re: [ftpext] COMB command IETF draft proposal

[Robert Oslin <[e-mail]>]

"Have you experienced any issues"

- yes, with larger (multi-GB) size files; however there are ways to combine the file so that timeouts can be avoided in most cases. I made a couple of fleeting references to those methods in the spec, but really it would be up to the server vendor to research and implement the optimal recombine technique so that timeouts could all but be eliminated for everything but exceedingly large files... and even that could be avoided by the client implementing logic that extends or temporarily ignores timeouts when waiting for the result of a COMB operation, perhaps extending the timeout proportional to the file size that was uploaded (same could apply for HASH operations on very large files). I can update the draft to make note of such cases and recommended workarounds; however I wouldn't make any mandatory requirements out of those. BTW, one possibility is for the server to return 200 OK once the operation is approved but not yet consummated, i.e. making it more of an asynchronous command, but that may lead to even more problems down the road (like if client decides to do something with the resulting file even though it has not yet been fully created/recombined).

Robert 

-----Original Message-----
From: Van Glass [mailto:[e-mail]] 
Sent: Wednesday, June 08, 2011 3:15 PM
To: Robert Oslin
Cc: Robert McMurray; [e-mail]
Subject: Re: [ftpext] COMB command IETF draft proposal

I think that having the server perform the delete tasks is preferred for the simple reason that this would reduce the number of commands going across the command channel.  We have not yet implemented COMB but would like to pursue it.  With that said I'm curious to know your experience COMBing a large number of files or very large files.  Have you experienced any issues where the COMBine process on the server side takes a long amount of time, possibly resulting in a command channel timeout on the client side?

Thanks,

Van Glass
JSCAPE
Managed File Transfer and Network Solutions http://www.jscape.com




On Wed, Jun 8, 2011 at 1:59 PM, Robert Oslin <[e-mail]> wrote:
> Thanks for the feedback! Issuing a hash (or XCRC until HASH is 
> formally ratified) makes a lot of sense and will be inserted as a 
> strongly recommended practice (i.e. SHOULD). As for the delete 
> operation, I would be careful to make it mandatory of the client, 
> rather leaving it open to interpretation, as there are many ways for 
> the server to execute the combine operation, which may or may not 
> result in the original parts being available for deletion. For example 
> a rudimentary concatenate operation by the server on ascii type files 
> (which would be quite slow) would result in each part being removed by 
> nature of the concatenation operation, hence a subsequent delete 
> operation would result in 5xx errors returned to the client on each 
> DELE request. Perhaps it would be wise to recommend that the client 
> check for parts post operation and do cleanup if any parts remain? In 
> our particular implementation (and in the handful of servers that 
> support COMB so far), it is the server that does
>  the cleanup as a function of the combine operation. Let me know if that makes sense, or if not, then how to deal with the myriad ways in which servers may go about combining files and how to deal with corner cases.
>
> Robert Oslin
>
> -----Original Message-----
> From: Robert McMurray [mailto:[e-mail]]
> Sent: Wednesday, June 08, 2011 2:11 PM
> To: Robert Oslin; [e-mail]
> Subject: RE: [ftpext] COMB command IETF draft proposal
>
> Thanks, Robert.
>
> I like the idea behind the COMB command; I could see where this could be quite useful for transferring large files.
>
> One consideration that I have in your draft is verbiage that "The server-FTP process SHOULD delete each of the parts identified by the COMB command once the target file is created." I would think that the cleanup behavior for partial files would be left to the client, since the remaining parts of the draft place the responsibility on the client for splitting the file and keeping track of the parts. For example:
>
>      C> STOR foo.txt
>      S> 226 Transfer complete.
>      C> STOR bar.txt
>      S> 226 Transfer complete.
>      C> COMB foobar.txt foo.txt bar.txt
>      S> 250 COMB command successful.
>      C> DELE foo.txt
>      S> 250 DELE command successful.
>      C> DELE bar.txt
>      S> 250 DELE command successful.
>
> As you mentioned in your draft, it would be advantageous if COMB was used with the HASH command to verify data integrity before the client takes care of the deletion tasks. For example:
>
>      C> OPTS HASH MD5
>      S> 200 MD5
>      C> STOR foo.txt
>      S> 226 Transfer complete.
>      C> STOR bar.txt
>      S> 226 Transfer complete.
>      C> COMB foobar.txt foo.txt bar.txt
>      S> 250 COMB command successful.
>      C> HASH foobar.txt
>      S> 213 MD5 426f62526f636b73 foobar.txt
>      C> DELE foo.txt
>      S> 250 DELE command successful.
>      C> DELE bar.txt
>      S> 250 DELE command successful.
>
> Thanks again.
>
> Robert McMurray
> [e-mail]
>
> ------------------------------
> From: Robert Oslin [mailto:[e-mail]]
> Sent: Wednesday, June 08, 2011 9:15 AM
> To: [e-mail]
> Subject: [ftpext] COMB command IETF draft proposal
>
> "Any submission to the IETF intended by the Contributor for publication as all or part of an IETF Internet-Draft or RFC and any statement made within the context of an IETF activity is considered an "IETF Contribution".
>
> It only took me 10+ years, but here's my contribution the FTP ext community (attached).
>
> Essentially an early draft to ratify the commonly used COMB command for support of multi-part (a.k.a accelerated) uploads. I welcome comments/questions from the community. Ideally redlining the document (or via comments in the doc). I will resubmit using formal IETF draft (in ASCII) once I've addressed all comments/questions in the MS Word version (hope that's ok and doesn't break and IETF rules).
>
> Thanks,
>
> Robert Oslin
> Director of Product Management
> Tel: [phone]
> Fax: [phone]
> Send me large files securely
>
> www.globalscape.com
>    (NYSE Amex:GSB) *This communication, including attachments, is for the exclusive use of the addressee and may contain proprietary, confidential or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this communication and destroy all copies.
>
>
> _______________________________________________
> ftpext mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ftpext
>