Re: [nfsv4] FW: I-D ACTION:draft-faibish-nfsv4-pnfs-access-permissions-check-03.txt

[<[e-mail]>]

Tom,

Thanks for taking a look at this draft in such a prompt fashion.  All
four of your concerns turned up areas that needs attention.  Here are
the short answers (I've paraphrased your concerns as questions):

1) Is an MDS required to remember which clients have declared what
devices to be inaccessible?

That was not the intention; "To the extent that an MDS can determine
..." was supposed to be the text that would allow an implementation to
make its own decisions about when/whether/how to keep track of client
inability to access storage devices.  It looks like some stronger
language is needed to make that clear. FWIW, "SHOULD NOT" was
deliberately used instead of "MUST NOT", but that wasn't the entire
story.

> Given that we need this mechanism for the client to report errors, how
then does the server
> know when it can start using these storage devices again? Even if the
MDS knows a positive
> change took place, it has to rely on the client to do the checking.

The previous version of the draft added a callback for this purpose, but
that callback has been removed as not realistic at this juncture.  Hence
the practical answers to the above restart question probably involve
administrative intervention.

2) Should NFS4ERR_PERM or NFS4ERR_ACCESS be used for access permission
denial?

It's NFS4ERR_ACCESS, mea culpa; this will be fixed in -04.  In addition,
the list of I/O retry errors (when a failed storage device I/O is
retried via the MDS) that require clients to return the layout is
probably also wrong well beyond NFS4ERR_PERM vs. NFS4ERR_ACCESS :-(.

3) What are the other errors [beyond PERM/ACCESS or NXIO] that a server
can see for a client-inaccessible-storage-device and how is the server
supposed to react to those errors?

The current draft allows any error, although some of them don't make
much sense.  NFS4ERR_IO is an obvious error that is needed, as I expect
most SCSI errors to be mapped to that error.  As for which error code is
used, the server is supposed to log the error, but I don't think there's
any server processing that depends on which error the client sent (did
you see something of concern?).  OTOH, as part of digging into which I/O
retry errors require layout return, the list of errors allowed for an
inaccessible data server could be generated and added to the next
version of the draft.

4) What do the error codes actually mean? 

We need NFS4ERR_NXIO for inaccessible devices, so that error would need
to be unobsoleted for this purpose. Beyond that, see the response to 3)
- it looks like an explicit list is needed of errors that the client is
allowed to use when returning a layout or layouts because a storage
device in inaccessible.

As for a client getting NFS4ERR_STALE from a data server/storage device,
that error could be sent to the MDS or mapped to NFS4ERR_NXIO. IMHO,
it's a much better idea to just pass NFS4ERR_STALE on to the MDS so that
the MDS knows *exactly* what went wrong.  OTOH, NFS4ERR_NXIO would be
appropriate for inability to contact the data server (e.g., no TCP
connection, or RPC service is non-responsive).

OTOH, I'd prefer not to take the lid off the Pandora's box of passing
through SCSI errors, as the info involved doesn't always propagate up
the SCSI stack.

I'm glad this is a "Draft" ;-).

Thanks,
--David


> -----Original Message-----
> From: Tom Haynes [mailto:[e-mail]]
> Sent: Monday, July 12, 2010 4:19 PM
> To: Black, David
> Cc: [e-mail]
> Subject: Re: [nfsv4] FW: I-D
ACTION:draft-faibish-nfsv4-pnfs-access-permissions-check-03.txt
> 
> [e-mail] wrote:
> > This is the revised permissions check draft - it actually deals with
any
> > circumstance under which a client cannot access a pNFS data server
and
> > wants to report that inaccessibility.
> >
> > Thanks,
> > --David
> >
> >
> 
> 1)  This:
> 
>    To the extent that an
>    MDS can determine whether storage devices are accessible to
clients,
>    an MDS SHOULD NOT include a storage device in any pNFS layouts sent
>    to a client that cannot access that storage device. At a minimum,
the
>    server SHOULD perform these storage device accessibility checks
>    before exporting a filesystem that supports pNFS and when the
device
>    configuration for such an exported filesystem is changed (e.g., to
>    add a storage device).
> 
> 
> implies to me that the MDS has to keep track of
> LAYOUT4_RET_REC_FSID_NO_ACCESS
> and LAYOUT4_RET_REC_FILE_NO_ACCESS layout return types per client.
I.e.,
> once it
> knows a client has problems with a specific storage device, it should
> avoid using that
> device again.
> 
> Given that we need this mechanism for the client to report errors, how
> then does the server
> know when it can start using these storage devices again? Even if the
> MDS knows a positive
> change took place, it has to rely on the client to do the checking.
> 
> Is this a difference between MUST and SHOULD? I.e., does having SHOULD
> mean that
> the MDS can hand out the storage devices again to see if the client
can
> suddenly start
> using them again?
> 
> 2) Is it NFS4ERR_PERM or NFS4ERR_ACCESS for access permission denial?
> 
>    o  NFS4ERR_PERM SHOULD be used for access permission denial; and
> 
> 
>  From 5661:
> 
> 15.1.6.1. NFS4ERR_ACCESS (Error Code 13)
> 
>    Indicates permission denied.  The caller does not have the correct
>    permission to perform the requested operation.  Contrast this with
>    NFS4ERR_PERM (Section 15.1.6.2), which restricts itself to owner or
>    privileged-user permission failures, and NFS4ERR_WRONG_CRED
>    (Section 15.1.6.4), which deals with appropriate permission to
delete
>    or modify transient objects based on the credentials of the user
that
>    created them.
> 
> 15.1.6.2. NFS4ERR_PERM (Error Code 1)
> 
>    Indicates requester is not the owner.  The operation was not
allowed
>    because the caller is neither a privileged user (root) nor the
owner
>    of the target of the operation.
> 
> Since this document talks about mount issues, I went back to RFC 1813
> and the MOUNT protocol. MNT can return MNT3ERR_ACCES if
> the client does not have access rights to the export.
> 
> I think NFS4ERR_ACCESS is more consistent with prior protocols
> than NFS4ERR_PERM.
> 
> Going back to this draft, in section 3.3, I see:
> 
>    There are two NO_ACCESS layoutreturn_type4 values that indicate
lack
>    of storage device access, LAYOUT4_RET_REC_FSID_NO_ACCESS and
>    LAYOUT4_RET_REC_FILE_NO_ACCESS.
> 
> and
> 
>    An NFS error (nfsstat4) is
>    included in the layoutreturn data structures for these two types to
>    distinguish access permission problems from device inaccessibility:
> 
> 
> I think access has been overloaded here and to clarify things,
> NFS4ERR_PERM is selected
> over NFS4ERR_ACCESS.
> 
> The only other reasons I can see for using NFS4ERR_PERM instead of
> NFS4ERR_ACCESS
> are related to security:
> 
> a) if the user credentials were insufficient, i.e., kerberized access
to
> the storage device failed.
> 
> b) Section 13.12 of 5661:
> 
>    If the metadata server would
>    deny a READ or WRITE operation on a file due to its ACL, mode
>    attribute, open access mode, open deny mode, mandatory byte-range
>    lock state, or any other attributes and state, the data server MUST
>    also deny the READ or WRITE operation.
> 
> Which seems to point out a need for error codes for:
> 
> a) No access granted (mount for files, block devices have other means)
> b) Permission denied for the operation.
> c) Permission denied because of security.
> 
> The difference between b) and c) is that b) is per fileid and c) is
per
> fsid. So a MDS could
> still use the storage device in the layout for b), but should avoid
> using it for c).
> 
> 3) I find:
> 
>    An NFS error (nfsstat4) is
>    included in the layoutreturn data structures for these two types to
>    distinguish access permission problems from device inaccessibility:
> 
>    o  NFS4ERR_PERM SHOULD be used for access permission denial; and
> 
>    o  NFS4ERR_NXIO SHOULD be used for inability to access a device.
> 
>    Other NFS errors MAY be used when they are appropriate. All uses of
>    these two layout return types that report errors SHOULD be logged
by
>    the client.
> 
> 
> to be under-specified. What are the other errors that a server can see
> and how is it
> supposed to react to those errors?
> 
> I'd like to see language about which errors are MANDATORY to be
> supported and which
> are OPTIONAL. I know I can read the above to see that there are only
two
> MANDATORY
> ones, but I can also read it to see that all are MANDATORY.
> 
> I don't want clients shoehorning every error code back into these two.
> And I do want clarification
> on what a server should do with the OPTIONAL codes. I.e. is it free to
> reuse those storage
> devices the next time that client asks for a layout?
> 
> 4)
> 
> What if the storage device A returns NFS4ERR_STALE to the client while
> storage device
> B returns NFS4_OK for an operation on the same layout? But for a
> different file with the
> same layout, it is given NFS4_OK from both DSs. This isn't necessarily
> either a permission issue nor
> a device inaccessibility issue. (It could be either: the export was
> changed or the filesystem
> indicated by the filehandle does not exist. It could also just mean
that
> the indicated file does
> not exist.)
> 
> Would this be where LAYOUT4_RET_REC_FILE_NO_ACCESS and NFS4ERR_NXIO is
> appropriate?
> 
> Which leads to an even more interesting question, what constitutes a
> NFS4ERR_NXIO error?
> As NFS4ERR_NXIO is defined by 5661 to not be a valid return code from
> any operation
> or CB, I would take it to mean simply that the storage device is not
> responding to the client.
> If the client got any error back from the storage device, then it
could
> not use NFS4ERR_NXIO.
> 
> As the NFS4ERR_STALE could simply mean that the filehandle refers to a
> file which doesn't
> exist, is it appropriate to inform the server to no longer use that
> storage device in the layouts
> assigned to that client?
> 
> Where this is going is perhaps now is a good time to add more
> informative error codes from
> the storage device to the client. This would in turn allow the client
to
> send these back to the
> MDS. I.e., there is a world of difference between this device (fsid)
> does not exist and this file
> does not exist.
>