Re: [ftpext] COMB command IETF draft proposal

[John C Klensin <[e-mail]>]

--On Monday, June 20, 2011 15:33 -0500 Robert Oslin
<[e-mail]> wrote:

>>> ASCII is a lot easier ...
> 
> Noted.
> 
>>> I usually take the position of an FTP purist whose
>>> relationship with the protocol goes all the way back to some
>>> of the original design meetings.
> 
> I greatly appreciate the academic "FTP purist" viewpoint. I am
> coming at this from a purely tactical and practical angle,
> i.e. we have millions of users and businesses worldwide using
> our FTP client and server products that are regularly
>...

Part of the question when one brings something to the IETF for
standardization is to try to understand what value is added by
the process.   I'm trying to get you to think about a few things
-- if you do, and the conclusion is "impractical given deployed
base", I promise to be unhappy but I can live with it.

>...
>>> (4) Remembering that we originally intended FTP to be
>>> asynchronous between the command and data streams, it may be
>>> too late now, but the FTP-ish way to do this probably would
>>> have been something like:
> 
>   CWD wherever-you-want-this-to-end-up
>   BOMPT   (Begin-Odd-Multipart-Transfer)
>   PASV  (or PORT, here and below)
>   STRU local-name
>   PASV
>   STRU local-name
>   PASV
>   STRU local-name
>   ...
>   COMBine remote-name
> 
> I don't think it is too late, given the early stages of this
> draft, but yes from a practical standpoint it would require
> that vendors who are using COMB in its current incarnation to
> update their products; however it may be possible to both
> improve upon the design and grandfather the current design.

That is really good news from my point of view.
 
>>> The STRU commands would presumably all get 2yz replies
>>> specifying the name used or failure codes permitting
>>> something else to be done.  The server would be required to
>>> return those STRU replies in order even if the transfers
>>> finished out of order.
> 
> Unfortunately my knowledge of the STRU command is quite
> limited. I don't see (in any RFC) where a local-name value can
> be provided as an argument. Perhaps you mean STOU (store
> unique), or is there a way in which STRU is used that I'm
> totally missing?

Nope.  I just demonstrated that I should never write from memory
when I'm tired and/or preoccupied with other things.  STOU was
intended.  My apologies.

>>> Note that not only specifies completely separate and
>>> parallel data connections (rather than maybe trying to share
>>> a single TCP data stream), but, by using initiation and
>>> competition commands and letting the server keep track of
>>> its own part names,  it completely eliminates the quotes,
>>> hoping the path syntax on the local or remote systems
>>> matches whatever you've assumed, and, by using STRU,
>>> eliminates both the need to be sure that the part names
>>> chosen on the sender machine are available on the receiver
>>> one and a small potential security problem.  It also permits
>>> use of REST, etc., on the individual streams if needed.
> 
> I agree with the concept of initiation and completion
> commands.

Obviously that requires introducing a new state into FTP.  At
one level, that is a big step.  At another... well, maybe not so
big and more cleanliness may be important.

> That was what I was referring to when I wrote in a
> prior message "I missed something very important in the draft
> that that will require a bit of draft re-work". I believe a
> preparatory command is needed in order to alert the server of
> the client's intent, although my motives are slightly
> different from yours: I wanted to provide the server with all
> necessary information needed to (optionally) employ advanced
> combine techniques, such as streaming the combine process vs.
> post transfer combine. Here is what I was going to propose,
> except that I failed to consider STOU, which appears superior
> to STOR in that STOU eliminates the need for temporary
> filenames managed by the client (instead, as you indicated,
> allowing the 2zy reply could include the temporary unique name
> managed by the server.

I haven't gone back and checked the spec today, but I think the
2yz reply to STOU is actually required to return the
generated/unique name.  So nothing new there.

> Example:
>   CWD wherever-you-want-this-to-end-up
>   BMPT   remote-name, number-of-parts, original-size-in-bytes
>   PASV  (or PORT, here and below)
>  STOU  local-name (chunk1)
>   PASV
>   STOU local-name (chunk2)
>   PASV
>   STOU local-name (chunk3)
>   ...
>   COMB remote-name
> 
> Here is how my proposed logic differs (but only slightly) from
> yours:
> 
> 1. Use of four letter command name for the prepare portion,
> call it BMPT, PREP, or whatever.

Of course.   You and I haven't worked together, but I'm
notoriously bad at picking names and so tend to use things in
examples and preliminary proposals that no sane person would
actually pick for real use :-)

> 2. The prepare command needs to provide sufficient information
> to the remote server so that advanced file combine techniques
> can be used, whether use of scratch or reserved temporary
> space, or sequence of blocks, or whatever. In our case (since
> we are thinking sparse files), the server needs to know the
> name of the destination file, the number of bytes, and total
> parts that will be transferred. With this information the
> server can deduce each of the part sizes that will
> subsequently be sent by the client, and therefore where the
> server should begin writing each byte offset within the sparse
> file. For other techniques there may be more or less
> information required; however I cannot think of any additional
> parameters aside from size and number of expected parts at the
> moment.

Interesting.  I need to think about this more, but my sense is
that your model would work well with some media in which it is
possible to write from several positions in a "file" at more or
less the same time and badly when that was not the case.  I
remember long ago having to worry about FTP transfers writing
directly to tape on the target systems, for which that would be
a terrible model and one would need to buffer, assemble, and
then write instead.   Perhaps we have reached the point where
purely sequential devices aren't worth worrying about any more,
but I'd like to think about it a bit.

> 3. STOU instead of STRU used (which is what I believe you
> meant).

yes

> 4. The one concern I have with NOT providing the
> client-managed individual part names in the preparatory
> command or in each STOR command is that it is possible for the
> order of the parts to be received out of sequence. Remember
> that the client will likely open up multiple data streams in
> parallel fashion (or nearly so), and the server might not
> handle them in the order received, hence the server would not
> know at which byte offset to begin writing each part received
> (i.e. is this part the beginning of the sparse file, somewhere
> in the middle, end of file, etc.). I believe the way to
> overcome this problem is for the client to not begin sending
> each subsequent part until it has received a 1xx reply from
> the server indicating that the server has accepted the STOU
> command, opened a data channel, and recognized it as part X of
> Y (Y being the number of parts communicated in the BMPT
> command). Once the 1xx reply is received the client would
> spawn a new thread and issue the next STOU command. Ideally
> (depending on latency and such), each part would be initiated
> within a second or two from each prior part and only after the
> initiation of the transfer has been recognized by the server.
> What I have NOT addressed in this email is problems such as a
> started MP transfer but not all parts are sent, aborted
> transfers, and inter-session transfers (started MP but lost
> connection and some parts still remain or were partially
> transferred).

Right.  The other slightly-problematic issue is when other FTP
commands change the format or encoding of the file on the source
system into a network-standard form, possibly to be converted to
something else on the target system.  If those conversions are
done in as part of the data stream reading or writing process,
rather than creating temporary intermediaries, notions like
"byte offsets" might become problematic... and things are sort
of downhill from there.   One could have that problem with any
TYPE other than Image, with any use of (the real) STRU, and
maybe in other cases.

On the other hand, note that a variation on the (actual) STRU
command (RFC 959, Section 3.1.2.3) can be used to specify the
parameters for paged, random-access, transfers.  Noting that it
requires a "page index" internal to each block, perhaps it would
be useful for your case rather than doing what first occurred to
me, which would have been adding an extra sequencing argument to
STOU.  (If you want to initiate a bunch of transfers in
parallel, having to queue them up behind 1yz acknowledgements
strikes me as a bad idea if it can be avoided). 
 
> 5. The other problem is when the client has multiple
> multi-part and standard (single-part) transfers initiated in
> parallel. I.e. what happens if I send two files using 8 parts
> and another smaller file using no parts (standard transfer) at
> the same time. This is a likely scenario, given current
> client's multi-threaded capabilities.  What we don't want is
> the server confusing transfers and assigning wrong parts to
> wrong files. Yes, keeping the STOU <filename> commands
> identical for each unique source filename makes a lot of
> sense, but how does the server know which BMPT command to
> associate with the subsequent STOU commands? Assuming
> parallelism but showing both logs combined, in chronological
> order:
> 
> BMPT  blah.dat, 4, 2235478
> BMPT  blah2.dat, 4, 2431234
> STOU A.dat
> STOU A.dat
> STOU B.dat
> STOU A.dat
> STOU B.dat
> STOU A.dat
> STOU B.dat
> STOU B.dat
> COMB blah.dat
> COMB blah2.dat

> Notice here that the server will have received sequential
> (back to back) BMPT (preparatory) commands, followed by
> back-to-back STOU commands. There needs to be a way for the
> server to identify which batch of STOU commands belong to
> which BMTP command.

And, of course, if one were trying to map a revised FTP state
model with that setup, things would get to be bad news indeed.
If there were no wait-for-next-command conditions at all, one
could recast that as 

BMPT  blah.dat, 4, 2235478
STOU A.dat    (or any other fragment identifiers one 
STOU A.dat    (liked )
STOU A.dat
STOU A.dat
COMB blah.dat
TPMB          (using a cousin of the "if... fi" convention)
BMPT  blah2.dat, 4, 2431234
STOU B.dat    (note that repeating A.dat and B.dat
STOU B.dat    (in this way could be a serious problem
STOU B.dat    (if they represented real file names 
STOU B.dat    (on the client.. but see below)
COMB blah2.dat
TPMB
SYNC          (wait for everything to complete on the
              (server side) 

That difference is partially dictated by trying to avoid nested
states (serial stacked states seem less problematic in the FTP
model) plus some distributed database experience that argues for
a a "commit and wait for everything at once" operation, applied
infrequently, rather than a lot of individual completion states.
Actually designing "SYNC" would require some rather careful
analysis of error replies and conditions (see below) but so
would COMB as you describe it above.

> I think the way to solve this problem is to require that the
> <Filename> value provided in the preparatory command (BMPT in
> this example) match the filename provided in the STOU commands
> (and vice versa). The user could always rename the file later
> if desired, e.g.
> 
> BMPT A.dat, 4, 2235478
> BMPT B.dat, 4, 2431234
> STOU A.dat
> STOU A.dat
> STOU B.dat
> STOU A.dat
> STOU B.dat
> STOU A.dat
> STOU B.dat
> STOU B.dat
> COMB A.dat
> COMB B.dat
> RNFR A.dat
> RNTO blah.dat
> RNFR B.dat
> RNTO blah2.dat

First, as mentioned above, this doesn't work well if the client
needs to build the fragments and keep them, however temporarily,
as real file names.   More to the point, if you are going to get
this far out into extension-space, nothing prevents you from
either inventing a new STOU-like command, or changing the syntax
of STOU if it occurs inside BMTP-state, to have an additional
transfer identifier.   So you would have, e.g., 

	BMPT foo, 4, 2235478
	BMPT bar, 4, 2431234
	STOU foo A1.dat
	STOU foo A2.dat
	STOU bar B1.dat
	STOU foo A3.dat
	   [...]
	COMB foo A.dat
	COMB bar B.dat

which strikes me as more flexible and a lot less error-prone.  I
like it less than some of the other suggestions here, but that
is partially just a matter of taste.

> Let me make one more point here: Depending on the combine
> techniques used bye server, the COMB command may not be of any
> use other than to indicate to the server that it (the client)
> is done transferring all parts, kind of like a "hey, I'm done
> now, did you get everything and combine ok?", to which the
> server would reply in the affirmative if for example the
> sparse file had filled out and all bytes were accounted for.
> If the server received the command but something was missing
> or there were others errors somewhere along the combine
> process, the server would reply with a transitive negative
> completion reply (hey, send me the last part again), or
> permanent negative completion reply (the bytes received don't
> match what the preparatory command said I was going to
> receive, therefore COMB failed"). 

Two observations, without suggesting specific remedies right now:

	(i) Go back and reread the discussion of PAGE-structured
	STRU in RFC 959.  We've been through parts of this
	before and there was actually some significant
	experience with the construct... including, if I recall,
	discovering that keeping the page index as a header for
	each transmission was a whole lot safer than anything
	involving keeping track of transmission initiation
	sequencing.

	It may not be exactly right for your purposes -- in
	particular, STRU does not assume multiple data channels
	running in parallel although I guess it could use them--
	but the idea of attaching sequencing/ locational headers
	to what is transmitted in the data channel(s) has some
	appeal.
	
	(ii) It is important to try to preserve the simplicity
	of FTP responses.  If possible, a client should be able
	to tell what to do from the codes alone.  If not, the
	response text should be sufficiently rigorously
	constructed so as to permit _really_ simple parsing and
	analysis.  It seems to me that some of your suggestion
	above would require a rather complex response structure
	and that is pretty scary.

> Likewise the COMB command
> could retain it's original syntax for backwards compatibility,
> i.e. if the server receives a COMB command that wasn't
> preceded by a preparatory BMPT command, then it will expect
> the COMB command to supply all necessary parameters (dst name,
> part1, part2, etc.) which means I still need to solve the
> naming/syntax issues.
> 
> Here is an example of a complete sequence (less CWD and
> PASV/PORT mode operations) that helps summarize what I've
> stated so far:
> 
> On Command Channel
> u> BMPT A.dat, 4, 2235478
> s> 200 OK, ready for MP transfer
> ...(wait for 2yz on the various data channels)
> u> COMB A.dat
> s> 200 Combine successful
> 
> On Data Channel 1
> u> STOU A.dat
> s> 150 Data connection opened for A.dat as part 1 of 4, ready
> for next part s> 226 Transfer complete

Whoops!  Maybe this is why I was confused about multiple  above.
As soon as you start sending commands down the data channels,
rather than the control one, you aren't doing FTP any more
because you are violating a few key assumptions of the protocol.
Note that STRU (again, the real one) deals with that by sending
STRU P (and then STOR or STOU) on the control connection and
then using special "page" headers on the data connection(s) to
keep things sorted out.

> On Data Channel 2
> u> STOU A.dat
> s> 150 Data connection opened for A.dat as part 2 of 4, ready
> for next part s> 226 Transfer complete
> 
> On Data Channel 3
> u> STOU A.dat
> s> 150 Data connection opened for A.dat as part 3 of 4, ready
> for next part s> 226 Transfer complete
> 
> On Data Channel 4
> u> STOU A.dat
> s> 150 Data connection opened for A.dat as part 4 of 4
> s> 226 Transfer complete  <--client waits for last part to get
> 226 reply then command channel sends COMB to confirm all is
> well and recombined.

>>> The use of STRU in that context would also permit an
>>> intelligent FTP-server to use some type of pool (or scratch
>>> or reserved temporary) space for the intermediate files,
>>> copying only the final, reassembled, file into the target
>>> directory.  That would eliminate several of the operational
>>> or security risks you discuss.  Arranging to disassemble the
>>> file into pool space on the client machine (or using the
>>> interleaved block approach discussed below) would eliminate
>>> a number of others.
> 
> Agreed, which is why sending a preparatory command is so
> important. We just need to make sure that all necessary
> information is provided so that intelligent FTP servers can
> use different techniques for combining the files, while making
> sure we avoid potential problemd associated with parallel
> multi-part transfers and out-of-sequence sending of parts.

Ack

>>> FWIW, the above model could work rather well in the download
>>> direction, elegantly eliminating the need for odd tricks
>>> with the checkpoint/restart mechanism.
> 
> Yes it could. Currently we use multiple REST commands at
> various offsets to accomplish this, but from an FTP purists
> perspective, that was ever the intended manner for REST to be
> used. I just don't know if I want to tackle MP downloads in
> this draft, given that this problem is already solved, albeit
> in a non-elegant manner.

Really a separate issue.  My point was only that, if one first
established the concept of a new state for bundled transfer
groups, it might later be applicable to other things.

>>> All of that said, experience with a number of
>>> distributed/parallel storage systems, including some
>>> distributed database update models and RAID striping, would
>>> suggest that the most transfer-efficient way to do what you
>>> are trying to accomplish would be to use a completely
>>> different model:  treat the file to be transferred as a
>>> sequence of blocks of specified size, open a bunch of
>>> connections, push block 1 onto connection 0, block 2 onto
>>> connection 1, and continue with each block, M, going out
>>> onto connection (M modulo number of connections).
> That would not only permit more optimization but would permit
> reassembly to start while the transfer was still in progress.
> 
> This is how sparse files work and why it is necessary for the
> client tell the server how many parts and total size of bytes
> (so that it can perform the M modulo number of connections
> calculation on the server side).
> http://en.wikipedia.org/wiki/Sparse_file

Not clear from your spec that this is what you intend to do.
More important, if you are going to stripe files on the network,
the model outlined above with separate file-pieces being
assembled on the client and then sent and reassembled is
hopelessly cumbersome because striping doesn't require
pre-structuring anything or even accurately knowing the size of
the file to be sent because one could rotate through stripe
segments dynamically.  Ideally one would probably use segments
whose sizes were calculated from the datagram size with sliding
windows on each channel.  It would imply no need for an explicit
recombination step and could be really fast.  But it is very
different.

>...
>>> (6) The first two paragraphs of your Security Considerations
>>> section are a little bogus.  Providing COMB support only to
>>> authorized users, etc., may protect against certain classes
>>> of malicious attacks, but they provide absolutely no
>>> protection against ignorance, stupidity, or carelessness,
>>> any of which can be easily exploited by an attacker as well
>>> as creating risks of equally-damaging accidents.
> 
> I think it goes without saying that we can never protect fully
> against user ignorance and stupidity. Heck, the DELE command
> is pretty efficient at destroying data as well. My notes in #6
> are simply the warning labels that remind you not to use your
> hairdryer in the bathtub. I.e. misuse (or abuse) of the COMB
> command can lead to file corruption, which is the same (or
> worse in some cases) than the destruction of data using DELE.
> I do think these warnings are necessary and appropriate, but
> may change them a bit to match the proposed logic changes
> (which are quite substantial).

Sorry... I was trying to make two slightly different points,
which were (i) that designing a command that is unnecessarily
fragile is a bad idea (some of the ideas above make it less
fragile) and (ii) there is a difference between problems
resulting from "plain" ignorance, stupidity, or carelessness and
situations where those same properties can easily be used by an
attacker.  Again, explicit "start of..." and "end of..."
state-establishing commands eliminate some of the problem.

As far as DELE is concerned, there is a very long history of FTP
servers being configured to let selected users upload files but
not delete or overwrite anything... for precisely that reason.
Having server-generated intermediate file names (potentially
stored at server-selected special locations) reduces that
problem significantly as well.

> Thanks John for your feedback. I wish I had thought of the
> necessary preparatory commands BEFORE submitting my first
> draft, but I suppose that is what the working group is for -
> to point out flaws in the design and ways to improve, even if
> it does mean wholesale changes are required.

Exactly.  See my comment about value-added above.

> I'll wait until I hear from others and if everyone is on board
> I'll float a revamped draft that includes an initiation
> (preparatory) and completion command sequence for MP uploads.

Ack.

best,
   john