Re: [nfsv4] seeking clarifications in server-side copy CB_OFFLOAD feature

[Olga Kornievskaia <[e-mail]>]

Thanks for the comments Bruce!

On Fri, Mar 3, 2017 at 1:48 PM, J. Bruce Fields <[e-mail]> wrote:
> On Thu, Mar 02, 2017 at 04:07:42PM -0500, Olga Kornievskaia wrote:
>> I'm seeking some clarification for doing the server-side copy feature.
>> EINVAL in read beyond the end of the file has been already raised in
>> another thread so I'll leave that aside. So here are a few more :
>> -- possible inconsistent behavior in presence of an error from doing
>> an async copy vs sync. CB_OFFLOAD can return an error and a partial
>> copy result but there is no mechanism to do so in synchronous COPY. If
>> CB_OFFLOAD returned an error and a partial copy, is that really an
>> error or should it be a short copy?
>
> On the client side, in the Linux case, the client is implementing
> copy_file_range(), which doesn't have the option of returning both an
> error and a number of bytes copied.  Returning an error to the
> application when the destination was actually modified is unacceptable.

Are you saying it is not acceptable to fail copy_file_range() and do a
partial copy or am I not reading it correctly? Why is this not
acceptable? Currently if I do a "cp /mnt/foo /mnt2/bar" and say there
is not enough space "cp" will display ENOSPC error but it will copy as
much as it could in /mnt/bar until it failed.

> Returning success and a short copy is OK, I think--if the error's
> important then the application will get it when it tries to copy the
> rest.
>
>> I can see a case where an error should not be ignore -- if error is
>> ENOSPC. Should the client if received an error in CB_OFFLOAD propagate
>> the error to the application?
>
> I think it's OK to ignore the ENOSPC.  The application will try to write
> the rest, the client will turn that into a new COPY request, this time
> the response will probably be 0 bytes copied and ENOSPC, so this time
> the client can return the error to the client.

Ok I agree about ENOSPC.

But, in general, 0 + some ERROR might not necessarily mean an error.
When the server rebooted and copied 0 bytes and maybe returned an
error (?), it should not be considered an error on the (NFS) client.

I'm actually hoping that server will return -1 and ENOSPC. But the
spec doesn't dictate that failure should return -1.

> Alternatively I guess the client could cache that ENOSPC result and
> return it on the next copy without talking to the server?  But then
> there's a risk that the client could miss the fact that space was freed
> up since the previous COPY.

ENOSPC is typically treated as a "permanent" error. By that I mean
even now in v4 WRITE can fail with ENOSPC, server could a sec later
freed space and if client send it again it would succeed but we fail
the operation.

I would have like the COPY to fail on 1st occurrence of ENOSPC
(instead of ignoring it) but I guess I'm fine to send it again and
then finally replying to application with an error.

>> Should it be server's responsibility to then if some
>> error occurred but a partial copy succeeded to then not return the
>> error to the client (that leads to the 2nd question).
>
> I'd rather servers didn't use this odd quirk of the protocol, but maybe
> it's too late to stop them.  And it seems easy enough just to ignore the
> error on the client side in the case bytes were copied.

I can argue that it's easier for the server to ignore the error due to
the reboot. There is no error to represent server reboot. There is no
wording in the spec to state that if CB_OFFLOAD.error = X it means
source server reboot. So if client got CB_OFFLOAD.error = X it is
really guessing if it means server rebooted or something else.

I guess I'm afraid that by ignoring all errors I'm not thinking of
some example where it was the wrong thing to do. Currently this is how
I've coded up the client to ignore the errors (all except ENOSPC which
I guess I can ignore as well let client got another ENOSPC) but I have
a nagging feeling I'm missing something.

>> -- related to errors returned in CB_OFFLOAD, there is no guidance with
>> regards to the reboot recovery. If the destination server unable to
>> finish the copy due to the source server rebooting, should it reply
>> back with a successful short copy or should it return some error (and
>> short copy of possibly 0bytes).
>
> A short copy sounds reasonable to me.

short copy with an error, right?

>
> --b.
>
>> To address these two questions I find some wording in the spec that
>> confuse me (section 15.2.3):
>>
>> "If a failure does occur for a synchronous copy, wr_count will be set
>> to the number of bytes copied to the destination file before the error
>> occurred." ... it also talks about consecutive bits there but then it
>> proceeds to also say that "If the failure occurred for an asynchronous
>> copy, .... It will be able to determine the bytes copied from the
>> coa_bytes_copied in the CB_OFFLOAD argument."
>>
>> Since it talks about "wr_count", then it must means that COPY's
>> nfsstat4 status is not an error. Does this then mean that if copy
>> "failed" then the error must be ignored on the server and a partial
>> copy results propagated to the client. But for the async copy,
>> "coa_bytes_copied" means that CB_OFFLOAD also set nfsstat4 to some
>> error. So error is not ignored in the async case?
>>
>> -- the spec lists the kind of errors CB_OFFLOAD reply can return but
>> there is no wording for what kind of errors be included in
>> CB_OFFLOAD's call. For instance, I can see that a source server reboot
>> can translate to ERR_BAD_HANDLE or it could be BAD_SESSION or
>> STALE_CLIENTID or BAD_STATEID. It is not clear what errors to check
>> for and recover by restarting the copy and what errors shouldn't
>> happen. Something like BAD_SESSION implies that server-to-server
>> protocol was NFS4.x x>=1. Can the client receive an NFSv3 error in
>> CB_OFFLOAD?
>>
>> While I realize that server-to-server copy details is not a part of
>> the RFC7862, but what the client receives in CB_OFFLOAD and how to
>> interpret that is unclear.
>>
>> Thank you.
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/nfsv4