Hi Marc,

On 9/21/07, Marc Boorshtein <mboorshtein@gmail.com> wrote:

> > Thats why many virtual
> > directories have dynamic group plugins to make dynamic groups work and
> > act like static groups.
> Yes this is one such approach but it scares me in terms of the size of
> entries that
> will be returned.  You're going to be caching a lot in memory.  Consider a
> dynamic
> group with 1M members.  You now have to stuff your entry to be returned with
> 1M
> values for the [unique]member attribute.  If the average size for a member's
> internationalized
> DN is 512 bytes then we're going to return an entry which is 512M+ in size.
> I don't want
> to keep this entry in memory as the server builds it to send it out the
> door.

If the application were pulling all members back then yes, this would
be a major issue.  It is not however a requirement to bring all the
members into the virtual directory and then return them to the client
to test a group membership.  Generally a group test will look like:

ldapsearch ... -b "cn=my group,ou=groups,dc=domain,dc=com" -s base
"(uniqueMember=cn=me,ou=people,dc=domain,dc=com") 1.1

Very true but I was specifically talking about reading the entire group.  But you're
right this would work as well as a Compare request which is even cheaper to perform
on most servers.
If the user is a member of the group then the group dn will be
returned as the only entry.  While there are definitely really badly
written applications that will retrieve the entire group and then do
an evaluation thats pretty rare.  

Yeah this is what I am afraid of.  I know of many apps like this unfortunately.  Some
are really big commercial applications yuk!

Even if it does happen you would
have the same problem with a static group.  

Yep this is the reason why static groups are useless when the set increases in size.

I've actualy deployed this
solution with Oracle's virtual directory where the static members were
150k-500k members (and is a 256mb entry any better then a 512mb entry?
:-D) and it worked very well.

Haa!  That's crazy. Try doing that under heavy load.  I'd like to see how the VD then handles OOM

I'm still not sure what the advantage of your solution is over a
dynamic group, but I'm probably missing something.

Hmmm are you up to date on the X.500 admin model?  SubtreeSpecifications and what not?
The whole point is to leverage the same concepts rather then the standard dynamic group
being based on the LDAP URL.  SS is much more flexible.

> I'd love to do that but for the reasons above I'm constrained.  The search
> control based
> approach is perhaps the best way to do this but then again the client has to
> get a bit
> smarter.  Perhaps both options can exist and when the size of the group is
> beyond some
> threshold the static group like lookup fails?

well, applications just aren't written well.  1/2 the commercial LDAP
applications out there have poorly written LDAP modules that were
built for a specific customer and were just 'released'.


> This get's me to another point.  LDAP needs a partial result return protocol
> response
> which could allow streaming large amounts of attributes back to clients to
> prevent
> excessive memory consumption.
> > The answer to 'why dont we do it' is that applications need to be able
> > to consume it.
> Yep this is the biggest problem but applications need to change and so does
> otherwise we're stuck doing stupid things like HTTP did with statelessness
> for decades.

Good luck to you there...I don't know that there will ever be an
LDAPv4 but I suppose thats another discussion.

As with everything it's up to the community.  I don't know either and it's probably a pipe dream
but oh what a good dream it is.

> Incidentally Authorization is already handled correctly by the X.500 model
> which is being
> specified in LDAP with the work being done by Steven Legg here:
>     http://tools.ietf.org/html/draft-legg-ldap-acm-bac-04
> Authorization does not need this after this draft becomes an RFC which shows
> a strong
> liklihood.  ApacheDS uses this same model - this is why we stuck so close to
> the X.500
> standards in our AC implementation.

Well, I'm not a huge X.500 fan but you say tomatoe....I'm curios to
check out the spec though.  Thanks for the pointer.

Well you're going to need to be.  I know reading X.500 specs make the best
ambien replacement in the world but LDAP is based on X.500 and is moving
closer to it.  IMO LDAP was too lightweight in an adverse reaction to the OSI
weight of X.500.  So now people realize we have to embrace X.500 concepts
and in particular the admin model.  Lookie here ..


There's much more coming down the pike.  I've been dreaming about the day
this would happen.  The admin model is going take the ad hoc crap out of LDAP
implementations today.  Standards are the only way to reach true interoperability
across these ideas here.

> Getting back to groups.  This proposed mechanism for dynamic groups has it's
> basis in X.500.
> The excellent points regarding client consumption that you sited can be
> ameliorated using some
> existing protocol features such as extended operations and search controls
> but we must explore
> the impact of them on clients and on the server.  I smell a potential draft
> for a new specification
> here.  Interested?

Always glad to be involved and bounce ideas around.  I think I take a
different tact but thats probably OK too.

Sure definitely.

Extended operations & controls are great and all but applications just
don't use them.  If I were to design my super group it would have 4

1.  Members can be dynamic
2.  Members can be static
3.  Members can be other groups
4.  I can test a group membership with a single ldap search call

I like these items here especially #3 which did not occur to me until now.  Also note you can use a compare
op too but you're right most apps don't have a freakin clue about how to do a compare either.   The name of
this game is to cater to the dumb clients already out there that won't change.

Now how to do this under the covers I bet you are far better at
answering then me.  Again I can do this with a virtual directory now
(MyVirtualDirectory has a DynamicGroups insert and a EmbeddedGroups
insert that does this).  Maybe ApaceDS can do it better because it has
all of its data locally (actually I'm positive it could).

We do have our class of problems but we'll solve them with time.  My whole angle with this stuff
is to do it while extending (through IETF) and complying with standards and expected semantics. 
ADS is trying to achive these feature carefully without being another ad hoc concoction to respond
to the market.  We want to outlast the fads.

In fact I am certain virtualization can have a solid theory behind it through the X.500 admin model.
I'm in the process of defining conceptually views for LDAP using the admin model and stored
procedures.  This however is future work.

I do say I'd be very interested to get the other open source ldap
proejects (opends, openldap, penrose & fedorads) opinions on the

Hmmm as you know I've been intricately involved with the Penrose peeps having helped them use
apacheds under the hood.  I withdrew from them for various reasons but one was because they
don't have a clue about the protocol and X.500.  Penrose is just an ad hoc concoction to make money
quick. Nothing wrong with that but it bastardizes the technology with one-offs jammed in to make
it to market.  You need a balance.

The other efforts are legitimate IMO but these folks are best engaged through an open specification