Re: [ftpext] COMB command IETF draft proposal

[Robert Oslin <[e-mail]>]

>>ASCII is a lot easier ...

Noted.

>>I usually take the position of an FTP purist whose relationship with the protocol goes all the way back to some of the original design meetings.

I greatly appreciate the academic "FTP purist" viewpoint. I am coming at this from a purely tactical and practical angle, i.e. we have millions of users and businesses worldwide using our FTP client and server products that are regularly performing multi-part uploads. My goal is to formalize (and potentially improve) the process so that other vendors can extend this existing benefit to their users, ideally in a non-proprietary fashion, much like our proprietary XCRC command which has been used for performing integrity checking for years and is now finally being formalized as the HASH command (albeit with many improvements) by Mr. Anthony Bryan, for the benefit of the entire FTP client/server vendor and user community.

>>I think you intend that COMB cause the transfer and then the combining function

Originally that was not my intent. Presently COMB is nothing more than an operation for combining a unique set of files. My original intent was for the client process to determine whether to perform a multi-part transfer and for the client to use its own programmatic logic to determine byte offsets and what not, and then issue multiple parallel STOR commands for each part over discreet data channels. As far as the server was concerned these were unique (whole) file uploads. Only the client knew that those uploads were actually parts of larger file or not. See notes on (4) for more information.

>>You need to be extremely careful with concepts like "file path" and "directory path".

Agreed.

>>(3) Your use of quotes is un-FTP-ish and your command makes assumptions about the syntax of a name that is not local to the working directory on the server-FTP.

Thanks. This may be a moot point though. See notes on (4)

>>Note that the second is seriously pathological given your "spaces not needed" rule.

Agreed.

>>(4) Remembering that we originally intended FTP to be asynchronous between the command and data streams, it may be too late now, but the FTP-ish way to do this probably would have been something like:

  CWD wherever-you-want-this-to-end-up
  BOMPT   (Begin-Odd-Multipart-Transfer)
  PASV  (or PORT, here and below)
  STRU local-name
  PASV
  STRU local-name
  PASV
  STRU local-name
  ...
  COMBine remote-name

I don't think it is too late, given the early stages of this draft, but yes from a practical standpoint it would require that vendors who are using COMB in its current incarnation to update their products; however it may be possible to both improve upon the design and grandfather the current design.

>>The STRU commands would presumably all get 2yz replies specifying the name used or failure codes permitting something else to be done.  The server would be required to return those STRU replies in order even if the transfers finished out of order.

Unfortunately my knowledge of the STRU command is quite limited. I don't see (in any RFC) where a local-name value can be provided as an argument. Perhaps you mean STOU (store unique), or is there a way in which STRU is used that I'm totally missing?

>>Note that not only specifies completely separate and parallel data connections (rather than maybe trying to share a single TCP data stream), but, by using initiation and competition commands and letting the server keep track of its own part names,  it completely eliminates the quotes, hoping the path syntax on the local or remote systems matches whatever you've assumed, and, by using STRU, eliminates both the need to be sure that the part names chosen on the sender machine are available on the receiver one and a small potential security problem.  It also permits use of REST, etc., on the individual streams if needed.

I agree with the concept of initiation and completion commands. That was what I was referring to when I wrote in a prior message "I missed something very important in the draft that that will require a bit of draft re-work". I believe a preparatory command is needed in order to alert the server of the client's intent, although my motives are slightly different from yours: I wanted to provide the server with all necessary information needed to (optionally) employ advanced combine techniques, such as streaming the combine process vs. post transfer combine. Here is what I was going to propose, except that I failed to consider STOU, which appears superior to STOR in that STOU eliminates the need for temporary filenames managed by the client (instead, as you indicated, allowing the 2zy reply could include the temporary unique name managed by the server.

Example:
  CWD wherever-you-want-this-to-end-up
  BMPT   remote-name, number-of-parts, original-size-in-bytes
  PASV  (or PORT, here and below)
 STOU  local-name (chunk1)
  PASV
  STOU local-name (chunk2)
  PASV
  STOU local-name (chunk3)
  ...
  COMB remote-name

Here is how my proposed logic differs (but only slightly) from yours:

1. Use of four letter command name for the prepare portion, call it BMPT, PREP, or whatever.

2. The prepare command needs to provide sufficient information to the remote server so that advanced file combine techniques can be used, whether use of scratch or reserved temporary space, or sequence of blocks, or whatever. In our case (since we are thinking sparse files), the server needs to know the name of the destination file, the number of bytes, and total parts that will be transferred. With this information the server can deduce each of the part sizes that will subsequently be sent by the client, and therefore where the server should begin writing each byte offset within the sparse file. For other techniques there may be more or less information required; however I cannot think of any additional parameters aside from size and number of expected parts at the moment.

3. STOU instead of STRU used (which is what I believe you meant).

4. The one concern I have with NOT providing the client-managed individual part names in the preparatory command or in each STOR command is that it is possible for the order of the parts to be received out of sequence. Remember that the client will likely open up multiple data streams in parallel fashion (or nearly so), and the server might not handle them in the order received, hence the server would not know at which byte offset to begin writing each part received (i.e. is this part the beginning of the sparse file, somewhere in the middle, end of file, etc.). I believe the way to overcome this problem is for the client to not begin sending each subsequent part until it has received a 1xx reply from the server indicating that the server has accepted the STOU command, opened a data channel, and recognized it as part X of Y (Y being the number of parts communicated in the BMPT command). Once the 1xx reply is received the client would spawn a new thread and issue the next STOU command. Ideally (depending on latency and such), each part would be initiated within a second or two from each prior part and only after the initiation of the transfer has been recognized by the server. What I have NOT addressed in this email is problems such as a started MP transfer but not all parts are sent, aborted transfers, and inter-session transfers (started MP but lost connection and some parts still remain or were partially transferred).

5. The other problem is when the client has multiple multi-part and standard (single-part) transfers initiated in parallel. I.e. what happens if I send two files using 8 parts and another smaller file using no parts (standard transfer) at the same time. This is a likely scenario, given current client's multi-threaded capabilities.  What we don't want is the server confusing transfers and assigning wrong parts to wrong files. Yes, keeping the STOU <filename> commands identical for each unique source filename makes a lot of sense, but how does the server know which BMPT command to associate with the subsequent STOU commands? Assuming parallelism but showing both logs combined, in chronological order:

BMPT  blah.dat, 4, 2235478
BMPT  blah2.dat, 4, 2431234
STOU A.dat
STOU A.dat
STOU B.dat
STOU A.dat
STOU B.dat
STOU A.dat
STOU B.dat
STOU B.dat
COMB blah.dat
COMB blah2.dat

Notice here that the server will have received sequential (back to back) BMPT (preparatory) commands, followed by back-to-back STOU commands. There needs to be a way for the server to identify which batch of STOU commands belong to which BMTP command.

I think the way to solve this problem is to require that the <Filename> value provided in the preparatory command (BMPT in this example) match the filename provided in the STOU commands (and vice versa). The user could always rename the file later if desired, e.g.

BMPT A.dat, 4, 2235478
BMPT B.dat, 4, 2431234
STOU A.dat
STOU A.dat
STOU B.dat
STOU A.dat
STOU B.dat
STOU A.dat
STOU B.dat
STOU B.dat
COMB A.dat
COMB B.dat
RNFR A.dat
RNTO blah.dat
RNFR B.dat
RNTO blah2.dat

Let me make one more point here: Depending on the combine techniques used bye server, the COMB command may not be of any use other than to indicate to the server that it (the client) is done transferring all parts, kind of like a "hey, I'm done now, did you get everything and combine ok?", to which the server would reply in the affirmative if for example the sparse file had filled out and all bytes were accounted for. If the server received the command but something was missing or there were others errors somewhere along the combine process, the server would reply with a transitive negative completion reply (hey, send me the last part again), or permanent negative completion reply (the bytes received don't match what the preparatory command said I was going to receive, therefore COMB failed"). Likewise the COMB command could retain it's original syntax for backwards compatibility, i.e. if the server receives a COMB command that wasn't preceded by a preparatory BMPT command, then it will expect the COMB command to supply all necessary parameters (dst name, part1, part2, etc.) which means I still need to solve the naming/syntax issues.

Here is an example of a complete sequence (less CWD and PASV/PORT mode operations) that helps summarize what I've stated so far:

On Command Channel
u> BMPT A.dat, 4, 2235478
s> 200 OK, ready for MP transfer
...(wait for 2yz on the various data channels)
u> COMB A.dat
s> 200 Combine successful

On Data Channel 1
u> STOU A.dat
s> 150 Data connection opened for A.dat as part 1 of 4, ready for next part
s> 226 Transfer complete

On Data Channel 2
u> STOU A.dat
s> 150 Data connection opened for A.dat as part 2 of 4, ready for next part
s> 226 Transfer complete

On Data Channel 3
u> STOU A.dat
s> 150 Data connection opened for A.dat as part 3 of 4, ready for next part
s> 226 Transfer complete

On Data Channel 4
u> STOU A.dat
s> 150 Data connection opened for A.dat as part 4 of 4
s> 226 Transfer complete  <--client waits for last part to get 226 reply then command channel sends COMB to confirm all is well and recombined.


>>The use of STRU in that context would also permit an intelligent FTP-server to use some type of pool (or scratch or reserved temporary) space for the intermediate files, copying only the final, reassembled, file into the target directory.  That would eliminate several of the operational or security risks you discuss.  Arranging to disassemble the file into pool space on the client machine (or using the interleaved block approach discussed below) would eliminate a number of others.

Agreed, which is why sending a preparatory command is so important. We just need to make sure that all necessary information is provided so that intelligent FTP servers can use different techniques for combining the files, while making sure we avoid potential problemd associated with parallel multi-part transfers and out-of-sequence sending of parts.

>>FWIW, the above model could work rather well in the download direction, elegantly eliminating the need for odd tricks with the checkpoint/restart mechanism.

Yes it could. Currently we use multiple REST commands at various offsets to accomplish this, but from an FTP purists perspective, that was ever the intended manner for REST to be used. I just don't know if I want to tackle MP downloads in this draft, given that this problem is already solved, albeit in a non-elegant manner.

>>All of that said, experience with a number of distributed/parallel storage systems, including some distributed database update models and RAID striping, would suggest that the most transfer-efficient way to do what you are trying to accomplish would be to use a completely different model:  treat the file to be transferred as a sequence of blocks of specified size, open a bunch of connections, push block 1 onto connection 0, block 2 onto connection 1, and continue with each block, M, going out onto connection (M modulo number of connections).
That would not only permit more optimization but would permit reassembly to start while the transfer was still in progress.

This is how sparse files work and why it is necessary for the client tell the server how many parts and total size of bytes (so that it can perform the M modulo number of connections calculation on the server side). http://en.wikipedia.org/wiki/Sparse_file


>>(5) Some of the material in 3.3 don't make any sense.  If the server "SHOULD" or "MAY" send the specified responses, it seems to me that is obligatory on you to specify what happens if the server chooses to do something else.  Presumably it is not open season for anything you don't specify -- I'd predict severe interoperability problems if a server responded to your 550 case by returning, e.g., 987.

Noted.

>>(6) The first two paragraphs of your Security Considerations section are a little bogus.  Providing COMB support only to authorized users, etc., may protect against certain classes of malicious attacks, but they provide absolutely no protection against ignorance, stupidity, or carelessness, any of which can be easily exploited by an attacker as well as creating risks of equally-damaging accidents.

I think it goes without saying that we can never protect fully against user ignorance and stupidity. Heck, the DELE command is pretty efficient at destroying data as well. My notes in #6 are simply the warning labels that remind you not to use your hairdryer in the bathtub. I.e. misuse (or abuse) of the COMB command can lead to file corruption, which is the same (or worse in some cases) than the destruction of data using DELE. I do think these warnings are necessary and appropriate, but may change them a bit to match the proposed logic changes (which are quite substantial).

Thanks John for your feedback. I wish I had thought of the necessary preparatory commands BEFORE submitting my first draft, but I suppose that is what the working group is for - to point out flaws in the design and ways to improve, even if it does mean wholesale changes are required.

I'll wait until I hear from others and if everyone is on board I'll float a revamped draft that includes an initiation (preparatory) and completion command sequence for MP uploads.

Robert