Re: [ldapext] DBIS - new IETF drafts

[Mark R Bannister <[e-mail]>]

On 06/01/2014 23:17, Simo wrote:
> On Mon, 2014-01-06 at 21:29 +0000, Mark R Bannister wrote:
>> On 06/01/2014 18:41, Simo wrote:
>>> I have to say I have to agree with Howard here, although I wouldn't
>>> consider rfc2307bis *the* solution, at least it tried to fix some of
>>> the most glaring issues ion rfc2307, there is lots that can be
>>> improved of course, but the direction is good.
>> As I wrote in my response to Howard just now, I considered both RFC2307
>> and the improvements made in RFC2307bis when I wrote the DBIS drafts.  I
>> have not disregarded lessons learned from the past, I am simply
>> attempting to move it onto the next logical step.  Look harder, I think
>> you'll see a lot of technical content you recognise (although admittedly
>> I couldn't keep much of the descriptive text because it didn't fit the
>> new model).
> One thing I forgot to say is that I haven't seen a good rationale
> document to be honest, what did you identify as problems, what did you
> see in the field, what solutions did you discard (you mentioned
> something about RFC2307bis not being good enough in reply to Howard) and
> why. What is the guiding principle behind the choices you made ?
> Apologies if you have mentioned all this, my quick read of the initial
> pages and introductory paragraphs didn't seem to give this information.

You're right I've not written a good rationale document yet and it's 
something I've been meaning to do.  The abstract in 
draft-bannister-dbis-mapping-02 goes some of the way, it doesn't go all 
of the way.  Thinking about this chronologically:

   * At one of my clients, NIS was being migrated into Active Directory 
using the RFC2307bis schema.  This client had circa 6,000 Linux hosts 
and 1,500 Solaris.  They had to use a commercial product to achieve 
this, even though their Linux and Solaris estate had full Open Source 
support for putting all of their NIS maps, including the automounter, 
into LDAP and even though they could have used Open Source tools to get 
the Kerberos authentication working.  I even assessed how it could be 
done with Open Source tooling and wrote about it in my blog which you 
are welcome to read at your leisure: 
http://technicalprose.blogspot.co.uk/2011/10/linux-integration-with-active-directory.html. 
However, they chose a commercial product because the job was so 
difficult, the Linux tooling so fragmented, and RFC2307bis alone was not 
going to help them with a big merger of two NIS domains with clashing 
UIDs and GIDs.  It was at this time that I first decided that RFC2307bis 
did not go far enough to help with the complexities of large 
corporations, even given the similarities in architecture the way in 
which NSS is configured on Solaris and Linux is considerably (and 
unnecessarily) different due to lack of standardisation at the client 
layer.  Unless Open Source tools were to be patched, it would not be 
possible to manage the merger of two NIS domains with clashing UIDs/GIDs 
without renumbering large numbers of user and group accounts.  I 
realised at this time the importance of having a standardised 
abstraction/rewriting layer above the data that would essentially allow 
the data to be fragmented across the DIT, using a different mix of 
attributes to define the fields required by NSS databases, and removing 
the need to perform complex and error-prone data merge and cleansing 
operations in favour of simply linking and transforming.

    * Another of my clients, with approx. 35,000 Linux hosts and 1,000 
Solaris, struggled in particular with case sensitivity.  In fact they 
had to come up with their own schema on top of RFC2307bis in order to 
make things case sensitive again (like they were in NIS).  It seemed 
ludicrous to me that RFC2307bis had made stuff case insensitive, and 
there didn't appear to be a good reason for doing so.  So to fix this 
problem, I couldn't just redefine the same attributes again, because 
that would make migration from RFC2307bis to DBIS really rather tricky 
and probably impossible for large deployments.  So wherever an attribute 
in RFC2307bis is defined as case insensitive, but needed to be case 
sensitive, a new attribute would be required.

    * Looking first at RFC2307bis and deciding how best to tackle these 
problems, I also realised that defining all of the NSS databases in a 
single document was not a good approach.  If I were to support an 
abstraction layer above the data itself, allowing the flexibility I 
wanted to glue maps together from different parts of the DIT, first I 
would need a standard framework.  This would also solve another thing I 
didn't like about current implementations, that NSS client 
configurations can get long, especially when you want to start remapping 
attribute names, are not always defined in a single configuration file, 
and differ from one OS to another.  If I started with a standard 
framework, where all of the information pertaining to the location and 
transformation of the data were in the DIT, it would also make it easier 
for administrators to make structural changes to this data over time 
without ever having to touch the client.  Where would I define this 
framework?  In configuration map objects, defined in 
draft-bannister-dbis-mapping-02.  It seemed to make natural sense to 
define the framework in its own document, and for it to be extensible so 
that even after the standard NSS databases had been described in this 
framework, vendors could come along in the future and plug-in their own 
databases (for example, Solaris has extra databases such as user_attr, 
auth_attr, prof_attr and exec_attr that I wouldn't want to define within 
DBIS at this time but nothing stops Oracle adding them).  It was taking 
shape itself: one document to describe the framework, then one document 
for each of the standard NSS databases.

    * Netgroups, urgh.  A third field that always seems redundant, and 
I've worked at clients where they have thousands of nested netgroups and 
it's a horrible mess.  So I had to do something about those.  Cleaning 
up their definition so there is no redundant data was step one, 
introducing netservices to help reduce the sheer number of them was step 
two.  I hope the net result will be to make them a lot more appealing 
and a lot easier to use.

> This said,
>
> What is the point of a new schema if you feel bound to serve unchanged
> legacy clients for example, aren't you pretty much shackled to RFC2307
> that way ?

So a new schema is required anyway, if problems such as case 
insensitivity and to be avoided.  There are other things I've fixed 
throughout the schema as well which I won't list here, but should be 
clear in my drafts.

Legacy NIS clients can't use RFC2307, only if they have already been 
configured with nss_ldap.  So, legacy NIS clients can either be migrated 
to use nss_dbis, or alternatively we can point them at a pseudo NIS 
server that back-ends to DBIS.

Clients already migrated to nss_ldap need to move over to nss_dbis 
first.  This can be done quickly and in stages, because DBIS 
configuration maps can be defined that point to the existing data in 
LDAP.  It should be as painless as setting up a few configuration maps, 
then deploying nss_dbis to the clients.  New NSS data, such as new user 
accounts and group accounts, can start to be populated in a different 
area in the DIT using the new schema, and referenced by a different set 
of configuration maps, which are received by clients filtered by 
netgroup.  Don't forget that a single client can use data from multiple 
places in the DIT using a mix of different schemas in order to piece 
together a single map.  Then, optional if you want the benefits of the 
new schema, we may choose to do a mass migration of data from RFC2307 
schema to DBIS schema and get rid of RFC2307 completely.  Except 
switching from nss_ldap to nss_dbis, all this done off-client and 
seamlessly.  The clients are none the wiser that the data is being moved 
around and transformed beneath them. So, no, not shackled in any way to 
RFC2307.

> OTOH, if you think you can change stuff then why stick to the NIS
> paradigm, which doesn't really map well at all to modern directories ?

If by this you are referring solely to shadow entries (password 
policies), then I think I've answered this question in my response to 
Howard.  Do you have examples of anything besides shadow entries that 
don't map well to modern directories?  I could do with some specifics to 
give you a meaningful response.

>>>> If you're going to all the trouble of storing data into a universally
>>>> accessible distributed database, you must store it in a canonical format, not
>>>> a particular OS-specific format as NIS/RFC2307 did.
>>> If there is a flawed model that should *not* be followed going forward that is NIS.
>>> And especially netgroups, please spare us from the disease of NIS
>>> netgroups, they should be buried, there are much better ways to group
>>> diverse objects.
>> Please provide a technical description of "disease" please, because it's
>> hard to do anything constructive with a statement like that. Like it or
>> not, there are very big installations out there using NIS or RFC2307 and
>> netgroups.  If netgroups have been built into the way a corporation
>> works, they'll possibly never be extracted.  Embrace and extend (without
>> the extinguish) if you'd like to bring those corporations with you and
>> make improvements for everyone.  Ignore them, and you get IPv6 all over
>> again.  I have taken netgroups and tried to take the "disease" out of
>> them, while providing a clear migration path, but was it the same
>> disease you were suffering?  I don't know.  Perhaps.  More detail please.
> Well the definition of netgroups simply makes little sense as it is in
> NIS and RFC2307. The fact you have triplets but then only 2 parameters
> at a time can be effectively used at once makes little sense and keeping
> that representation simply maintains a bad data representation model and
> keeps legitimizing it instead of fading it off.

So this is why I've changed that.  I've slightly redefined the meaning 
of the "domain" component in the triplet, and I'm defining netgroup 
members using two multi-valued attributes: netgroupHost and 
netgroupUser, rather than by a triplet.  The domain component can 
optionally be used with either of those two attributes, and has a more 
useful meaning when applied.  Take a closer look at 
draft-bannister-dbis-netgroup-01 again.

>>>> A lot of the data elements and behaviors that the DBIS spec defines appear to
>>>> be client-implementation-specific details. It seems to me their definition is
>>>> inappropriate in (outside the scope of) a universally accessible distributed
>>>> data context.
>>> Same concerns here.
>> As before, I'm not sure I understand what you mean.  Please provide
>> specific examples.
> It looks like you are trying to build a glorified NIS, trying to follow
> closely the way information is stored in files, but why ?
> I do understand quite well the need for compatibility on the client, esp
> with nsswitch interfaces, but that doesn't mean the best representation
> in the directory is a straight copy, which is unfortunately what the
> NIS/RFC2307 schema extenders did.
>
> What I mean, and suspect Howard may mean too, is that what should be
> done is to provide the objects and attributes that matter to identity
> management today, and only as a second layer explain how a client can
> map those attributes to the legacy NIS concepts for clients that need to
> serve that representation internally through its legacy interfaces.
>
> This is what I ave tried to do in the FreeIPA project for example,
> although within the limits of what we could do many years ago when we
> started, and only later we provided a compatibility tree that provides a
> view of the data to make legacy clients happy, performing whatever
> transformations are necessary to make them work, but not binding the
> directory model to the legacy objects and attributes.

So the problem here is that I wanted a solution we can use right now, 
today, and with minimal pain.  You'll see from some of my description 
above that it's actually very easy to swap a client from nss_ldap to 
nss_dbis to gain lots of benefits.  Changing the data model considerably 
makes it a lot more difficult to migrate from one solution to another, 
and large corporations obviously want to minimise the amount of big 
change going on, as large changes are prone to error.  Migrating to DBIS 
involves a small number of steps which are all easily reversed.  Up-take 
depends on ease of migration, if you want a solution to be used you have 
to consider backwards compatibility, otherwise like I said before, you 
end up with IPv6 (which despite being invented in the 1990s I have 
personally yet to see any of my clients making any use of it).  I'm not 
someone who likes to design something "academically perfect" if no one 
besides academics are going to use it.

>> CRYPT is a throwback to RFC2307bis and provides a place to start. That's
>> why these are drafts, because they are a work in progress.  I need to
>> support old clients as well as new clients, possibly old clients that
>> cannot be upgraded.
> Old clients that can't be updated can't make any use of new schema nor
> follow it anyway, so I fail to see the point to cater for those clients
> in this schema. I am not saying you shouldn't make allowances to
> *optionally* support exposing those hashes for compatibility purposes,
> but it should be clear it is an option and there should be language in
> the security considerations about how to properly limit exposure of such
> hashes through ACI or other configuration etc...

I agree.  I really dislike having to support CRYPT.  If I could get rid 
of it completely I would.  But I would only get rid of it completely if 
I could still guarantee a smooth migration from one system to another.  
I don't see how that is possible.  You're going to force 10,000+ users 
to change their passwords?  Ok, users might not be such an issue because 
they are probably being forced to change their passwords every 90 days 
anyway.  But service accounts, that's very difficult.  I have to support 
CRYPT because I don't want a migration to DBIS forcing passwords to be 
changed.  However, I entirely agree that we should give people as much 
help as possible to migrate to a more secure password scheme, and should 
strongly recommend it.  If it's really easy to do, then it'll be a 
no-brainer.

>
>> Backwards compatibility is important.
> The only way to be backward compatible at the LDAP level is to maintain
> all the problems of RFC2307, which I would rather see left behind.

Indeed.  But as you'll see from my earlier points, you can use the 
RFC2307 schema with a DBIS client, and that can be a very useful 
stepping stone in moving on to a better schema.

>
>> If DBIS  doesn't support CRYPT, how do I support the old clients?
> Through a new PAM module ? After all if you can't even put a new pam
> module on those client syou most probably can't put a new nsswitch
> client either so the new schema will not be used at all.

I think I've covered CRYPT already.

>> Suggestions welcomed.
> The way I solved it in most cases is by using pam_krb5 (or equivalents),
> ie move auth completely way from LDAP for those clients.

Service accounts.  Don't want to force password changes.  Discussed this 
already above.

>> Now I was considering implementing pseudo NIS servers that
>> act as a gateway for NIS clients that can't be upgraded,
> We've done this too in FreeIPA (see slapi-nis directory module), because
> you have no choice after all, if you need to serve legacy clients you
> are better off providing a gateway, so you do not have to touch those
> clients at all, except when it is finally time to decommission them and
> then you can flip a single switch and throw away all the compatibility
> garbage you were forced to provide. Meanwhile the main DIT is free(er)
> to evolve.

Fine if migration occurs once and once only.  Large corporations buy 
companies.  Mergers & acquisitions mean that your data will be polluted 
again.  Rather than making each and every merger painful, with DBIS you 
have the flexibility to plug-in new data, even though it doesn't comply 
with the standards you prefer, and get using it straight away.  Then you 
can perform migrations under the covers in your own time without senior 
management breathing down your neck asking you when they can finally 
kill the legacy network links or legacy infrastructure because it's 
costing them lots of money to maintain it.

>
>>   but the PAM
>> modules on the old clients are going to expect passwords in CRYPT format
>> I assume?
> Well, the point of the "Pluggable Authentication Module" system,
> usually, is that you can plug new modules. I do understand there may be
> resistance doing that on some old systems, but you need to draw a line
> at some point.

Why draw a line?  We don't need to draw a line.  Old NIS clients can use 
DBIS if you point them at a NIS gateway server.  No changes required at 
all on the NIS client.  Old infrastructure can continue to rot for a few 
more years.  Sometimes we have no choice.  Senior management says "jump" 
and we have to jump.  "These 100 hosts are critical to the running of 
this bank and must NOT be touched under any circumstances".  Oh and also 
"decommission those old NIS servers".  DBIS with a NIS gateway allows us 
to do both.  Prior to DBIS, we could not meet both of these requirements.

>
>>   I had yet to prove this in a lab environment, but until now I
>> assumed that CRYPT would have to be the lowest common denominator, I'd
>> have to support it.  Better authentication schemes can be added as and
>> when I found the time to further explore the options.
> Exposing hashes should be a last resort for compatibility reasons only
> and should be disabled by default with appropriate ACIs.

I don't suggest DBIS makes any mention of ACIs.  That's up to local 
rules & procedures, not something that could possibly be standardised.

> My point of view is that using LDAP binds SHOULD be mandated for
> authentication for any new client following any new schema, and exposing
> hashes MUST be disabled by default, and explicitly enabled by admins
> that needs backwards compatibility. We need to move up the security bar,
> and you do that only with appropriate defaults, and new IETF work should
> reflect that IMHO.
>
> Simo.
>
>
>

I don't think LDAP binds can be mandated, we can only strongly recommend 
it.  Given the migration path I have already described, I find it highly 
unlikely that people will be able to move to LDAP binds and move off 
CRYPT immediately, I would rather encourage them to get onto DBIS first 
and then gently steer them down a more secure route afterwards.  To be 
honest, people can make up their own minds about security, we don't have 
to make their minds up for them.  We offer them the tools and the 
options, the interoperability, and they decide how secure they want to 
be.  Both of the clients I mentioned earlier, with 7,000 hosts and 
36,000 hosts respectively, used Kerberos authentication with RFC2307bis 
anyway.  So they had already elected a stronger password scheme.

Best regards,
Mark.