directory-api mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiran Ayyagari <kayyag...@apache.org>
Subject Re: Proper use of LdapConnectionPool
Date Mon, 02 Feb 2015 23:49:14 GMT
On Tue, Feb 3, 2015 at 7:40 AM, Harris, Christopher P <
chris_harris@baxter.com> wrote:

> Right.  I understand the difference between the scopes.
>
> The problem is that the only scope that works for me against AD, in Apache
> Directory Studio and the LDAP API, is SUBTREE.  The other 2 scopes do not
> return results.
>
can you try with JNDI and see if you are able to fetch the RootDSE. See
this[1] thread
for a possible caveat while using JNDI.

Let us know if this works, then we need to check why it is not returned for
the client using API

[1] https://community.oracle.com/thread/1157742

>
>  - Chris
>
>
> -----Original Message-----
> From: Kiran Ayyagari [mailto:kayyagari@apache.org]
> Sent: Monday, February 02, 2015 5:30 PM
> To: api@directory.apache.org
> Subject: Re: Proper use of LdapConnectionPool
>
> On Wed, Jan 28, 2015 at 8:27 AM, Harris, Christopher P <
> chris_harris@baxter.com> wrote:
>
> > O.K.  I think I remember why I stopped looking at the SearchScope.  If
> > I just change the scope to either ONELEVEL or OBJECT, the cursor comes
> > back empty.
> >
> > Is this an AD "feature" or is this a bug in the API?
> >
> > OBJECT scope on rootDSE should work, and looks like AD allows it as
> > well
>
> > It looks like I'll need to do 1 query using the SUBTREE scope.
> >
> no, this will not solve the problem, you _must_ do ONE_LEVEL searches
> otherwise you either reach time or size limits enforced by AD.
>
> get rootDSE first, read the 'namingContexts' and then perform your search
> on each context separately, recursively using ONE level scope.
>
>
> >  - Chris
> >
> > -----Original Message-----
> > From: Harris, Christopher P
> > Sent: Tuesday, January 27, 2015 6:16 PM
> > To: api@directory.apache.org
> > Subject: RE: Proper use of LdapConnectionPool
> >
> > Ah, crap.  I forgot to look at the Scope.  I've been using this code
> > for so long for single-search queries that I took it for granted.
> >
> > I'll try setting it to ONE_LEVEL to simply bask in the glory of the
> > speedy results, but still, doing just 1 query makes a hell of a lot more
> sense.
> >
> > Sorry, I don't know where my head was.
> >
> > Thanks for steering me down the right path.
> >
> >  - Chris
> >
> >
> > -----Original Message-----
> > From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
> > Sent: Tuesday, January 27, 2015 5:43 PM
> > To: api@directory.apache.org
> > Subject: Re: Proper use of LdapConnectionPool
> >
> > Le 27/01/15 23:07, Harris, Christopher P a écrit :
> > > Hi, Emmanuel.
> > >
> > > "Can you tell us how you do that ? Ie, are you using a plain new
> > connection for each thread you spawn ?"
> > > Sure.  I can tell you how I am implementing a multi-threaded
> > > approach to
> > read all of LDAP/AD into memory.  I'll do the next best thing...paste
> > my code at the end of my response.
> > >
> > >
> > > "In any case, the TimeOut is the default LDapConnection timeout (30
> > seconds) :"
> > > Yes, I noticed mention of the default timeout in your User Guide.
> > >
> > >
> > > "You have to set the LdapConnectionConfig timeout for all the
> > > created
> > connections to use it. there is a setTimeout() method for that which
> > has been added in 1.0.0-M28."
> > > When visiting your site while seeking to explore connection pool
> > options, I noticed that you recently released M28 and fixed DIRAPI-217
> > and decided to update my pom.xml to M28 and test out the
> > PoolableLdapConnectionFactory.  Great job, btw.  Keep up the good work!
> > >
> > > Oh, and your example needs to be updated to using
> > DefaultPoolableLdapConnectionFactory instead of
> > PoolableLdapConnectionFactory.
> > >
> > >
> > > "config.setTimeOut( whatever fits you );"
> > > Very good to know.  Thank you!
> > >
> > >
> > > "It is the right way."
> > > Sweeeeeeet!
> > >
> > >
> > > "Side note : you may face various problems when pulling everything
> > > from an AD server. Typically, the AD config might not let you pull
> > > more than
> > > 1000 entries, as there is a hard limit you need to change on AD if
> > > you
> > want to get more entries.
> > >
> > > Otherwise, the approach - ie, using multiple threads - might seems
> > > good,
> > but the benefit is limited. Pulling entries from the server is fast,
> > you should be able to get tens of thousands per second with one single
> thread.
> > I'm not sure how AD support concurrent searches anyway. Last, not
> > least, it's likely that AD does not allow more than a certain number
> > of concurrent threads to run, which might lead to contention at some
> point."
> > >
> > > Ah, this is why I wanted to reach out to you guys.  You guys know
> > > this
> > kind of in-depth information about LDAP and AD.  So, I may adapt my
> > code to a single-thread then.  I can live with that.  I need to pull
> > about 40k-60k entries, so 10's of thousands of entries per second
> > works for me.  I may need to run the code by you then if I go with a
> > single-threaded approach and need to check if I'm going about it in the
> most efficient manner.
> >
> > The pb with the multi-threaded approach is that you *have* to know
> > which entry has children, because they won't give you such an info. So
> > you will end doing a search for every single entry you get at one
> > level, with scope ONE_LEVEL, and most of the time, you will just get
> > teh entry itself. That would more than double the time it takes to grab
> everything.
> >
> > >
> > >
> > >
> > > And now time for some code...
> > >
> > > import java.io.IOException;
> > > import java.util.Iterator;
> > > import java.util.List;
> > > import java.util.Map;
> > > import java.util.concurrent.ConcurrentHashMap;
> > > import java.util.concurrent.ExecutorService;
> > > import java.util.concurrent.Executors; import
> > > java.util.concurrent.TimeUnit; import java.util.logging.Level;
> > > import java.util.logging.Logger;
> > >
> > > import org.apache.commons.pool.impl.GenericObjectPool;
> > > import org.apache.directory.api.ldap.model.cursor.CursorException;
> > > import org.apache.directory.api.ldap.model.cursor.SearchCursor;
> > > import org.apache.directory.api.ldap.model.entry.Entry;
> > > import org.apache.directory.api.ldap.model.exception.LdapException;
> > > import org.apache.directory.api.ldap.model.message.Response;
> > > import org.apache.directory.api.ldap.model.message.SearchRequest;
> > > import
> > > org.apache.directory.api.ldap.model.message.SearchRequestImpl;
> > > import
> > > org.apache.directory.api.ldap.model.message.SearchResultEntry;
> > > import org.apache.directory.api.ldap.model.message.SearchScope;
> > > import org.apache.directory.api.ldap.model.name.Dn;
> > > import
> > > org.apache.directory.ldap.client.api.DefaultLdapConnectionFactory;
> > > import org.apache.directory.ldap.client.api.LdapConnection;
> > > import org.apache.directory.ldap.client.api.LdapConnectionConfig;
> > > import org.apache.directory.ldap.client.api.LdapConnectionPool;
> > > import org.apache.directory.ldap.client.api.LdapNetworkConnection;
> > > import
> > > org.apache.directory.ldap.client.api.DefaultPoolableLdapConnectionFa
> > > ct
> > > ory; import
> > > org.apache.directory.ldap.client.api.ValidatingPoolableLdapConnectio
> > > nF actory; import
> > > org.apache.directory.ldap.client.api.SearchCursorImpl;
> > > import org.apache.directory.ldap.client.template.EntryMapper;
> > > import
> > > org.apache.directory.ldap.client.template.LdapConnectionTemplate;
> > >
> > > /**
> > >  * @author Chris Harris
> > >  *
> > >  */
> > > public class LdapClient {
> > >
> > >       public LdapClient() {
> > >
> > >       }
> > >
> > >       public Person searchLdapForCeo() {
> > >               return this.searchLdapUsingHybridApproach(ceoQuery);
> > >       }
> > >
> > >       public Map<String, Person> buildLdapMap() {
> > >               SearchCursor cursor = new SearchCursorImpl(null,
> > > 300000,
> > TimeUnit.SECONDS);
> > >               LdapConnection connection = new
> > LdapNetworkConnection(host, port);
> > >               connection.setTimeOut(300000);
> > >               Entry entry = null;
> > >
> > >               try {
> > >                       connection.bind(dn, pwd);
> > >
> >  LdapClient.recursivelyGetLdapDirectReports(connection, cursor, entry,
> > ceoQuery);
> > >                               System.out.println("Finished all Ldap
> > > Map
> > Builder threads...");
> > >                       } catch (LdapException ex) {
> > >
> >  Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null,
> > ex);
> > >                       } catch (CursorException ex) {
> > >
> >  Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null,
> > ex);
> > >                       } finally {
> > >                               cursor.close();
> > >                                try {
> > >                                       connection.close();
> > >                               } catch (IOException ex) {
> > >
> >  Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null,
> > ex);
> > >                               }
> > >                       }
> > >
> > >               return concurrentPersonMap;
> > >       }
> > >
> > >       private static Person
> > recursivelyGetLdapDirectReports(LdapConnection connection,
> > SearchCursor cursor, Entry entry, String query)
> > >                       throws CursorException {
> > >               Person p = null;
> > >                       EntryMapper<Person> em =
> > > Person.getEntryMapper();
> > >
> > >               try {
> > >                               SearchRequest sr = new
> SearchRequestImpl();
> > >                               sr.setBase(new Dn(searchBase));
> > >                               StringBuilder sb = new
> > StringBuilder(query);
> > >                               sr.setFilter(sb.toString());
> > >                               sr.setScope( SearchScope.SUBTREE );
> >
> > Ahhhhh !!!! STOP !!!
> >
> > Ok, no need to go any further in your code.
> >
> > You are doing a SUBTREE search on *every single entry* you are pulling
> > from the base. if you have 40 000 entries, you will do something like
> > O(
> > 40 000! ) (factorial) searches. No wonder why you get timeout...
> > Imagine you have such a tree :
> >
> > root
> >   A1
> >     B1
> >       C1
> >       C2
> >     B2
> >       C3
> >       C4
> >   A2
> >     B3
> >       C5
> >       C6
> >     B4
> >       C7
> >       C8
> >
> > The search on root with pull A1, A2, B1, B2, B3, B4, C1..8 (14 entries
> > -> 14 searches)
> > Then the search on A1 will pull B1, C1, C2, B2, C3, C4 (6 entries -> 6
> > searches)
> > Then the search on A2 will pull B3, C5, C6, B7, C8, C9 (6 entries -> 6
> > searches)
> > Then the search on B1 will pull C1, C2 ( 2 entries -> 2 searches, *4 =
> > 8 ...
> >
> > At the end, you have done 1 + 14 + 12 + 8 = 35 searches, when you have
> > only 15 entries...
> >
> > If you want to see what your algorithm is doing, just do a search
> > using a SearchScope.ONE_LEVEL instead. You will only do somehow O(40
> > 000) searches, which is way less than what you are doing.
> >
> > But anyway, doing a search on the root with a SUBTREE scope will be
> > way faster, because you will do only one single search.
> >
> >
> > The information transmitted is intended only for the person(s) or
> > entity to which it is addressed and may contain confidential and/or
> > legally privileged material. Delivery of this message to any person
> > other than the intended recipient(s) is not intended in any way to
> > waive privilege or confidentiality. Any review, retransmission,
> > dissemination or other use of, or taking of any action in reliance
> > upon, this information by entities other than the intended recipient
> > is prohibited. If you receive this in error, please contact the sender
> and delete the material from any computer.
> >
> > For Translation:
> >
> > http://www.baxter.com/email_disclaimer
> > The information transmitted is intended only for the person(s) or
> > entity to which it is addressed and may contain confidential and/or
> > legally privileged material. Delivery of this message to any person
> > other than the intended recipient(s) is not intended in any way to
> > waive privilege or confidentiality. Any review, retransmission,
> > dissemination or other use of, or taking of any action in reliance
> > upon, this information by entities other than the intended recipient
> > is prohibited. If you receive this in error, please contact the sender
> and delete the material from any computer.
> >
> > For Translation:
> >
> > http://www.baxter.com/email_disclaimer
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
> The information transmitted is intended only for the person(s) or entity
> to which it is addressed and may contain confidential and/or legally
> privileged material. Delivery of this message to any person other than the
> intended recipient(s) is not intended in any way to waive privilege or
> confidentiality. Any review, retransmission, dissemination or other use of,
> or taking of any action in reliance upon, this information by entities
> other than the intended recipient is prohibited. If you receive this in
> error, please contact the sender and delete the material from any computer.
>
> For Translation:
>
> http://www.baxter.com/email_disclaimer
>



-- 
Kiran Ayyagari
http://keydap.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message