Return-Path: X-Original-To: apmail-directory-users-archive@www.apache.org Delivered-To: apmail-directory-users-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 76F869CD4 for ; Wed, 11 Apr 2012 18:46:12 +0000 (UTC) Received: (qmail 34917 invoked by uid 500); 11 Apr 2012 18:46:12 -0000 Delivered-To: apmail-directory-users-archive@directory.apache.org Received: (qmail 34898 invoked by uid 500); 11 Apr 2012 18:46:12 -0000 Mailing-List: contact users-help@directory.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@directory.apache.org Delivered-To: mailing list users@directory.apache.org Received: (qmail 34886 invoked by uid 99); 11 Apr 2012 18:46:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Apr 2012 18:46:12 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [62.153.167.38] (HELO postserver.ibs-ag.de) (62.153.167.38) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Apr 2012 18:46:07 +0000 Received: from postserver.ibs-ag.de (localhost [127.0.0.1]) by postserver.ibs-ag.de (Postfix) with ESMTP id E4C41A0E06; Wed, 11 Apr 2012 20:45:51 +0200 (CEST) Received: from IBSCAX2.ibs-ag.com (unknown [192.168.14.12]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by postserver.ibs-ag.de (Postfix) with ESMTPS id E34AFA0E04; Wed, 11 Apr 2012 20:45:51 +0200 (CEST) Received: from IBSMBX.ibs-ag.com ([fe80::78e3:9c63:57ee:7d20]) by IBSCAX2.ibs-ag.com ([192.168.14.12]) with mapi; Wed, 11 Apr 2012 20:45:46 +0200 From: To: , Date: Wed, 11 Apr 2012 20:45:44 +0200 Subject: RE: Plea for help with search performance Thread-Topic: Plea for help with search performance Thread-Index: Ac0YEuFWpUmd0M1zRSa2md9fBZyjcQAABMGA Message-ID: <2BE7E81B77921F43A6A273C2DF2FA6A43C4CC0E661@IBSMBX.ibs-ag.com> References: <2BE7E81B77921F43A6A273C2DF2FA6A43C4CC0E5AB@IBSMBX.ibs-ag.com> <2BE7E81B77921F43A6A273C2DF2FA6A43C4CC0E620@IBSMBX.ibs-ag.com> <4F85BC7E.2070406@gmail.com> <2BE7E81B77921F43A6A273C2DF2FA6A43C4CC0E64B@IBSMBX.ibs-ag.com> In-Reply-To: Accept-Language: en-US, de-DE Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, de-DE Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org OK thanks!! I included the text of the attachments in my response to you so I don't nee= d to attach. Trying to get somewhere with this tonight. Is there a quick fail safe way t= o create an index? I noticed in my partition directory there are no .db file for the attribute= s I've indexed. I think your hunch is correct. If it has to wait until tomo= rrow This is ok. Thanks for all your help!! Regards, Carlo Accorsi -----Original Message----- From: Emmanuel Lecharny [mailto:elecharny@apache.org] Sent: Wednesday, April 11, 2012 2:43 PM To: users@directory.apache.org Subject: RE: Plea for help with search performance ok. i'll check that tomorrow. the brst you can do is to fill a JIRA wjere y= ou cam attach files. be aware that those infos will be public. Le 11 avr. 2012 19:34, a =E9crit : > Emmanuel ok great.. I just responded to Kiran and realized the > attachments don't come through. > > I create everything with ldif files, restarting in between them. > I start with a clean install, star the default instance once, the > import the first file. > > ## begin file 1## > #version: 1 > > dn: > ads-partitionId=3Dcpro,ou=3Dpartitions,ads-directoryServiceId=3Ddefault,o= u=3Dc > onfig > objectclass: top > objectClass: ads-base > objectclass: ads-partition > objectclass: ads-jdbmPartition > ads-indexes: apacheRdn > ads-indexes: apacheSubLevel > ads-indexes: apachePresence > ads-indexes: apacheOneLevel > ads-indexes: apacheOneAlias > ads-indexes: apacheSubAlias > ads-indexes: apacheAlias > ads-indexes: entryUUID > ads-indexes: entryCSN > ads-indexes: o > ads-indexes: krb5PrincipalName > ads-indexes: objectClass > ads-indexes: ou > ads-indexes: uid > ads-indexes: employeeNumber > ads-indexes: displayName > ads-indexes: cn > ads-indexes: mail > ads-partitionSuffix: o=3Dcpro > ads-jdbmpartitionoptimizerenabled: TRUE > ads-partitioncachesize: 50000 > ads-partitionsynconwrite: TRUE > ads-partitionid: cpro > ads-enabled: TRUE > > dn: > ou=3Dindexes,ads-partitionId=3Dcpro,ou=3Dpartitions,ads-directoryServiceI= d=3Dd > efault,ou=3Dconfig > objectclass: organizationalUnit > objectclass: top > ou: indexes > > dn: > ads-indexAttributeId=3DapacheRdn,ou=3Dindexes,ads-partitionId=3Dcpro,ou= =3Dpart > itions,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: apacheRdn > ads-enabled: TRUE > ads-indexcachesize: 5000 > > dn: > ads-indexAttributeId=3DapacheSubLevel,ou=3Dindexes,ads-partitionId=3Dcpro= ,ou > =3Dpartitions,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: apacheSubLevel > ads-enabled: TRUE > ads-indexcachesize: 5000 > > dn: > ads-indexAttributeId=3DapachePresence,ou=3Dindexes,ads-partitionId=3Dcpro= ,ou > =3Dpartitions,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: apachePresence > ads-enabled: TRUE > ads-indexcachesize: 100 > > dn: > ads-indexAttributeId=3DapacheOneLevel,ou=3Dindexes,ads-partitionId=3Dcpro= ,ou > =3Dpartitions,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: apacheOneLevel > ads-enabled: TRUE > ads-indexcachesize: 100 > > dn: > ads-indexAttributeId=3DapacheOneAlias,ou=3Dindexes,ads-partitionId=3Dcpro= ,ou > =3Dpartitions,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: apacheOneAlias > ads-enabled: TRUE > ads-indexcachesize: 5000 > > dn: > ads-indexAttributeId=3DapacheSubAlias,ou=3Dindexes,ads-partitionId=3Dcpro= ,ou > =3Dpartitions,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: apacheSubAlias > ads-enabled: TRUE > ads-indexcachesize: 100 > > dn: > ads-indexAttributeId=3DapacheAlias,ou=3Dindexes,ads-partitionId=3Dcpro,ou= =3Dpa > rtitions,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: apacheAlias > ads-enabled: TRUE > ads-indexcachesize: 100 > > dn: > ads-indexAttributeId=3Do,ou=3Dindexes,ads-partitionId=3Dcpro,ou=3Dpartiti= ons,a > ds-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: o > ads-enabled: TRUE > ads-indexcachesize: 100 > > dn: > ads-indexAttributeId=3Dkrb5PrincipalName,ou=3Dindexes,ads-partitionId=3Dc= pro > ,ou=3Dpartitions,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: krb5PrincipalName > ads-enabled: TRUE > ads-indexcachesize: 100 > > dn: > ads-indexAttributeId=3DobjectClass,ou=3Dindexes,ads-partitionId=3Dcpro,ou= =3Dpa > rtitions,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: objectClass > ads-enabled: TRUE > ads-indexcachesize: 50000 > > dn: > ads-indexAttributeId=3DentryUUID,ou=3Dindexes,ads-partitionId=3Dcpro,ou= =3Dpart > itions,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: entryUUID > ads-enabled: TRUE > ads-indexcachesize: 5000 > > dn: > ads-indexAttributeId=3DentryCSN,ou=3Dindexes,ads-partitionId=3Dcpro,ou=3D= parti > tions,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: entryCSN > ads-enabled: TRUE > ads-indexcachesize: 5000 > > dn: > ads-indexAttributeId=3Dou,ou=3Dindexes,ads-partitionId=3Dcpro,ou=3Dpartit= ions, > ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: ou > ads-enabled: TRUE > ads-indexcachesize: 100 > > dn: > ads-indexAttributeId=3Duid,ou=3Dindexes,ads-partitionId=3Dcpro,ou=3Dparti= tions > ,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: uid > ads-enabled: TRUE > ads-indexcachesize: 50000 > > dn: > ads-indexAttributeId=3DemployeeNumber,ou=3Dindexes,ads-partitionId=3Dcpro= ,ou > =3Dpartitions,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: employeeNumber > ads-enabled: TRUE > ads-indexcachesize: 50000 > > dn: > ads-indexAttributeId=3DdisplayName,ou=3Dindexes,ads-partitionId=3Dcpro,ou= =3Dpa > rtitions,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-jdbmIndex > objectclass: ads-base > objectclass: top > ads-indexattributeid: displayName > ads-enabled: TRUE > ads-indexcachesize: 50000 > > dn: > ads-indexAttributeId=3Dcn,ou=3Dindexes,ads-partitionId=3Dcpro,ou=3Dpartit= ions, > ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-base > objectclass: ads-jdbmIndex > objectclass: top > ads-indexattributeid: cn > ads-enabled: TRUE > ads-indexcachesize: 50000 > > dn: > ads-indexAttributeId=3Dmail,ou=3Dindexes,ads-partitionId=3Dcpro,ou=3Dpart= ition > s,ads-directoryServiceId=3Ddefault,ou=3Dconfig > objectclass: ads-index > objectclass: ads-base > objectclass: ads-jdbmIndex > objectclass: top > ads-indexattributeid: mail > ads-enabled: TRUE > ads-indexcachesize: 50000 > > #Change password policy to allow hashed passwords from ldif > dn: > ads-pwdId=3Ddefault,ou=3DpasswordPolicies,ads-interceptorId=3Dauthenticat= ion > Interceptor,ou=3Dinterceptors,ads-directoryServiceId=3Ddefault,ou=3Dconfi= g > changetype: modify > replace: ads-pwdcheckquality > ads-pwdcheckquality: 0 > > ## end file 1 ### > > Then I restart the server and re connect > > ## begin file 2 ## > #version: 1 > > dn: o=3Dcpro > objectClass: organization > objectClass: top > o: cpro > > dn: ou=3Dint,o=3Dcpro > objectClass: organizationalUnit > objectClass: top > ou: int > description: Internal > > dn: ou=3Dusers,ou=3Dint,o=3Dcpro > objectClass: organizationalUnit > objectClass: top > ou: users > description: Internal Users > > dn: ou=3Dgroups,ou=3Dint,o=3Dcpro > objectClass: organizationalUnit > objectClass: top > ou: groups > description: Internal Groups > > dn: ou=3Dext,o=3Dcpro > objectClass: organizationalUnit > objectClass: top > ou: ext > description: External > > dn: ou=3Dusers,ou=3Dext,o=3Dcpro > objectClass: organizationalUnit > objectClass: top > ou: users > description: External Users > > #External Password policy > dn: > ads-pwdId=3Dcproext,ou=3DpasswordPolicies,ads-interceptorId=3Dauthenticat= ion > Interceptor,ou=3Dinterceptors,ads-directoryServiceId=3Ddefault,ou=3Dconfi= g > objectclass: top > objectclass: ads-base > objectclass: ads-passwordPolicy > ads-pwdattribute: userPassword > ads-pwdid: cproext > ads-enabled: TRUE > ads-pwdallowuserchange: TRUE > ads-pwdcheckquality: 0 > ads-pwdexpirewarning: 600 > ads-pwdfailurecountinterval: 30 > ads-pwdgraceauthnlimit: 5 > ads-pwdgraceexpire: 0 > ads-pwdinhistory: 5 > ads-pwdlockout: TRUE > ads-pwdlockoutduration: 0 > ads-pwdmaxage: 0 > ads-pwdmaxdelay: 0 > ads-pwdmaxfailure: 5 > ads-pwdmaxidle: 0 > ads-pwdmaxlength: 0 > ads-pwdminage: 0 > ads-pwdmindelay: 0 > ads-pwdminlength: 5 > ads-pwdmustchange: FALSE > ads-pwdsafemodify: FALSE > > dn: > ads-pwdId=3Dcproint,ou=3DpasswordPolicies,ads-interceptorId=3Dauthenticat= ion > Interceptor,ou=3Dinterceptors,ads-directoryServiceId=3Ddefault,ou=3Dconfi= g > objectclass: ads-passwordPolicy > objectclass: ads-base > objectclass: top > ads-pwdattribute: userPassword > ads-pwdid: cproint > ads-enabled: TRUE > ads-pwdallowuserchange: TRUE > ads-pwdcheckquality: 0 > ads-pwdexpirewarning: 10 > ads-pwdfailurecountinterval: 30 > ads-pwdgraceauthnlimit: 5 > ads-pwdgraceexpire: 0 > ads-pwdinhistory: 5 > ads-pwdlockout: TRUE > ads-pwdlockoutduration: 0 > ads-pwdmaxage: 60 > ads-pwdmaxdelay: 0 > ads-pwdmaxfailure: 5 > ads-pwdmaxidle: 0 > ads-pwdmaxlength: 0 > ads-pwdminage: 0 > ads-pwdmindelay: 0 > ads-pwdminlength: 10 > ads-pwdmustchange: FALSE > ads-pwdsafemodify: FALSE > > ## end file 2 ## > > I then update the partition cache to 50000 and restart again , before > importing any users. > Thanks. > > Regards, > Carlo Accorsi > > > > -----Original Message----- > From: Emmanuel L=E9charny [mailto:elecharny@gmail.com] > Sent: Wednesday, April 11, 2012 1:17 PM > To: users@directory.apache.org > Subject: Re: Plea for help with search performance > > Ok, > > after some investigation, I'm pretty sure that the indexes are not > created. When I create the indexes, I get those results : > > - 80000 entries injected into the server, with uid=3D > - displayName is indexed, uid is not indexed > > - first search : > connection.search("dc=3Dexample,dc=3Dcom", "(displayName=3D12345Awg-Rosli= , > Awg-Abd-Rahim SMDS-UIA/G/MMO52D)", SearchScope.SUBTREE, "*" ); > > result : > Delta search : 16 ms > entry found : > dn[n]: uid=3D12345,dc=3Dexample,dc=3Dcom > objectclass: organizationalPerson > objectclass: person > objectclass: inetOrgPerson > objectclass: top > title: Snr Operations Technician (D) > uid: 12345 > description: UI - S > businesscategory: Ops MDS (Malaysia) Sdn Bhd > departmentnumber: SMDS - UIA/G/MMO52D > employeenumber: A-A-R.Awg-Rosli > givenname: Awg-Abd-Rahim > cn: Awg-Rosli, Awg-Abd-Rahim SMDS-UIA/G/MMO52D > sn: Awg-Rosli > telephonenumber: 555-1212 > mail: A-A-R.Awg-Rosli@acme.com > displayname: 12345Awg-Rosli, Awg-Abd-Rahim SMDS-UIA/G/MMO52D > > - Second search : > connection.search("dc=3Dexample,dc=3Dcom", "(displayName=3D34567*)", > SearchScope.SUBTREE, "*" ); > > result : > Delta search substring : 9 ms > entry found : > dn[n]: uid=3D34567,dc=3Dexample,dc=3Dcom > objectclass: organizationalPerson > objectclass: person > objectclass: inetOrgPerson > objectclass: top > title: Snr Operations Technician (D) > uid: 34567 > description: UI - S > businesscategory: Ops MDS (Malaysia) Sdn Bhd > departmentnumber: SMDS - UIA/G/MMO52D > employeenumber: A-A-R.Awg-Rosli > givenname: Awg-Abd-Rahim > cn: Awg-Rosli, Awg-Abd-Rahim SMDS-UIA/G/MMO52D > sn: Awg-Rosli > telephonenumber: 555-1212 > mail: A-A-R.Awg-Rosli@acme.com > displayname: 34567Awg-Rosli, Awg-Abd-Rahim SMDS-UIA/G/MMO52D > > -third search : > connection.search("dc=3Dexample,dc=3Dcom", "(uid=3D67890)", > SearchScope.SUBTREE, "*" ); > > result : > Delta search no index : 38985 ms > > entry found : > dn[n]: uid=3D67890,dc=3Dexample,dc=3Dcom > objectclass: organizationalPerson > objectclass: person > objectclass: inetOrgPerson > objectclass: top > title: Snr Operations Technician (D) > uid: 67890 > description: UI - S > businesscategory: Ops MDS (Malaysia) Sdn Bhd > departmentnumber: SMDS - UIA/G/MMO52D > employeenumber: A-A-R.Awg-Rosli > givenname: Awg-Abd-Rahim > cn: Awg-Rosli, Awg-Abd-Rahim SMDS-UIA/G/MMO52D > sn: Awg-Rosli > telephonenumber: 555-1212 > mail: A-A-R.Awg-Rosli@acme.com > displayname: 67890Awg-Rosli, Awg-Abd-Rahim SMDS-UIA/G/MMO52D > > > So if the index is properly set, the search takes milliseconds to > complete (which is expected). Otherwise, it can take tens of seconds... > > Two possibilities : > - you don't initialize the index correctly. Please provide your > configuration and if you are using ADS embedded, please provide the > code that you use to embed it > - we have an issue in the way we initialize the index, which is more > likely to be the problem, as I fixed one issue in this area 2 weeks > ago (in > M7-SNAPSHOT) and I just fixed another one while doing the experiment... > > I'd like to rule out the first hypothesis first, but in any case, I'll > do some more check tonite or tomorrow. > > > Many thanks for your patience. > > > -- > Regards, > Cordialement, > Emmanuel L=E9charny > www.iktek.com > >