Re: [nfsv4] Proposal for end-of-life for fedfs-utils development

[Chuck Lever <[e-mail]>]

> On Jun 14, 2017, at 5:57 PM, Mkrtchyan, Tigran <[e-mail]> wrote:
> 
> Hi Chuck,
> 
> ----- Original Message -----
>> From: "Chuck Lever" <[e-mail]>
>> To: "J. Bruce Fields" <[e-mail]>
>> Cc: "fedfs-utils Developers" <[e-mail]>, "linux-nfs" <[e-mail]>, "NFSv4"
>> <[e-mail]>
>> Sent: Wednesday, June 7, 2017 8:02:19 PM
>> Subject: Re: [nfsv4] Proposal for end-of-life for fedfs-utils development
> 
>> Bruce, that's a good question, and an answer is worth sharing with
>> the nfsv4 WG mailing list, whom I've cc'd.
>> 
>> 
>>> On Jun 7, 2017, at 11:55 AM, [e-mail] wrote:
>>> 
>>> So if it's not too depressing I'd be curious what went wrong--did this
>>> turn out to be harder than we thought to get stable, or are people happy
>>> enough with automounting, or did we just not do a good job of explaining
>>> it to people that might use it, or some combination of all those?
>> 
>> If you're interested in an intriguing discussion of what makes a
>> successful protocol, I recommend RFC 5218. Now, not in any particular
>> order, the main reasons FedFS has not been widely adopted are (IMO of
>> course):
>> 
>> 
>> 1. Lack of vendor adoption
>> 
>> After the specifications were completed, we anticipated two independent
>> implementations. For various reasons the Solaris FedFS implementation
>> was abandoned. One big reason was LDAP.
>> 
>> NFS/Ganesha has expressed some interest in FedFS over the years, but
>> I'm not aware of an implementation.
>> 
>> NetApp abandoned FedFS before it was published as RFCs.
>> 
>> So we were left with just the Linux implementation, just like what
>> happened with SPKM.
>> 
>> 
>> 2. LDAP is onerously complex
>> 
>> The LDAP components of the Linux implementation were the worst to
>> implement by far. This also proved to be the case in the Solaris
>> prototype implementation. OpenLDAP is designed for massive scalability
>> but aggressively shuns ease of administration.
>> 
>> It was suggested that we integrate with FreeIPA, which is a Linux-based
>> management suite that can provide an LDAP service, a KDC, and a
>> certificate authority. But there was never enough user inertia to make
>> that effort.
>> 
>> There was resistance to integrating FedFS directly into nfs-utils
>> because the LDAP components would have added complex library
>> dependencies.
>> 
>> The LDAP pieces of FedFS are specified to use TLS, but NFS operates
>> using GSS and Kerberos. Getting these two worlds to work together was
>> going to be the next step (and also, figuring out how to make the LDAP
>> service use GSS instead, which we should have done before completing
>> the standard).
>> 
>> However, by the time the FedFS standards were complete, there was no
>> longer WG interest to address its shortcomings. There were two small
>> I-Ds published to start down that path. They went nowhere because the
>> IETF's pool of LDAP expertise is difficult to consult, now that
>> LDAP-related WGs have been disbanded.
>> 
>> IMO LDAP, which was chosen early in the 2000s by the IBM prototypes
>> that were to become FedFS, was ultimately a poor choice for what
>> eventually became the public FedFS standard. This can be attributed
>> to changing times.
>> 
>> 
>> 3. Lack of consensus about how to store junctions
>> 
>> There was never consensus in the Linux community about how to represent
>> junction objects on disk. NFS wanted something that would look naturally
>> like an empty directory to NFS versions that did not support referrals.
>> Samba wanted something that could behave like symbolic links, as it
>> had been using for its own DFS-style referrals. The filesystem folks
>> were not interested in creating a new distinct object that could fill
>> both roles.
>> 
>> As you are probably aware, Bruce, I asked about this every year at
>> LFS/MM for several years. I was always told "get back to us when you
>> have users."
>> 
>> Solaris went for reparse points in its implementation. Those are also
>> supported by FreeBSD. I think RPs would have been a great direction
>> but sadly these are not being actively pursued in the Linux FS
>> community.
>> 
>> Lack of user adoption sapped the energy out of the effort to find a
>> consensus. Though, if FedFS had been widely adopted before a consensus
>> was reached on junction object representation, we'd have a significant
>> data conversion problem.
>> 
>> 
>> 4. Existing implementations are capable enough
>> 
>> This is mostly speculation on my part, but FedFS was a competitive
>> response to the global namespace capabilities of AFS and SMB, not to
>> any particular stated need in NFS communities.
>> 
>> I've discussed the use of FedFS with various large enterprises, but
>> quite often the underlying needs are able to be filled by some form
>> of pNFS. I think this class of solution is what NetApp and Primary
>> Data have adopted.
>> 
>> Or put another way, pNFS seems capable of doing most anything that a
>> referral-based mechanism can do. And in non-NFSv4 environments, the
>> automounter is good enough for most people.
>> 
>> Certainly we could design something from the ground up that addresses
>> many of these shortcomings, but I get about one query about FedFS a
>> year. I just wonder if it would be worth the effort.
> 
> We  at DESY was interested in FedFS for two main reasons: one is to replace
> AFS and second is to build a federation on independent NFS server.
> The second option is not covered by pNFS as FedFS allows to run independent
> pNFS instances with different administrative domains, implementations and 
> namespaces, but still provide a common mount point. NetApp or Primary Data
> does not provide solution for that. Even we don't :-D.

The last time I visited DESY, I left with the impression that
there was a requirement that directories could be populated
programmatically (ie. one directory entry could represent a
group of files, perhaps on disparate NFS servers, and something
on each client determined which member of that group would be
opened). FedFS can't do that, but perhaps pNFS could.


> Why we have failed? There was not clear how to set it up. LDAP management
> was a mess. Schema was not standardized

The schema is now standardized in RFC 7532, but the pub date
for that document is recent (March 2015).


> and not available with existing LDAP server.

I recognized this could be an impediment to adoption, and
contacted the people responsible for 389ds and the OpenLDAP
community. I was told they could include the schema when
the schema was standardized and fedfs-utils had users.

fedfs-utils provides the schema in two forms, IIRC, ready
to copy to your favorite LDAP server.


> Currently we have 5 pNFS based instances with ~15PB in total. Some
> nodes have direct mounts. But most of 'generic' clients automunter config
> to fake FedFS as we don't know in advance which server will be used. Does
> it works - Yes, does it solves all problems - No.
> 
> I would like to see a federated name space. We have the demand, but may be it's
> only us.

I can't speak authoritatively about that, but DESY is the
only contact I've had about fedfs-utils in years.


> For non NFS world we have a HTTP federation. It uses HTTP redirects, similar to
> referrals. However, the tendency is to move towards distributed systems instead
> of federated ones. If this project survive, then we can attempt to have a second
> look at NFS federation. Life is a spiral.....

There's no plan to take down the upstream fedfs-utils
source code repository.


> Regards,
>   Tigran.
> 
>> 
>> 
>>> --b.
>>> 
>>> On Fri, Jun 02, 2017 at 11:01:05AM -0400, Chuck Lever wrote:
>>>> Upstream fedfs-utils has not been under active development for
>>>> two years or more, and there is a scant user base. I'd like to
>>>> propose making 0.10 the final major release of fedfs-utils.
>>>> 
>>>> The plan:
>>>> 
>>>> - Since 0.10 is in at least one major enterprise distribution,
>>>> I will remain available to integrate security fixes and make
>>>> new minor releases in the 0.10 line, as needed, for one to
>>>> two more years.
>>>> 
>>>> - Retire and remove fedfs-utils from upstream mirror distros
>>>> such as Fedora rawhide.
>>>> 
>>>> - Transfer utilities such as nfsref into nfs-utils, with
>>>> support for FedFS junctions removed.
>>>> 
>>>> - Announcements of the change in status will be made on
>>>> fedfs-utils-announce and on the wiki.linux-nfs.org site.
>> 
>> --
>> Chuck Lever
>> 
>> 
>> 
>> _______________________________________________
>> nfsv4 mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/nfsv4
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [e-mail]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Chuck Lever