directory-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Carlo.Acco...@ibs-ag.com>
Subject RE: Plea for help with search performance
Date Wed, 11 Apr 2012 18:45:44 GMT
OK thanks!!
I included the text of the attachments in my response to you so I don't need to attach.
Trying to get somewhere with this tonight. Is there a quick fail safe way to create an index?
I noticed in my partition directory there are no .db file for the attributes I've indexed.
I think your hunch is correct. If it has to wait until tomorrow
This is ok. Thanks for all your help!!

Regards,
Carlo Accorsi


-----Original Message-----
From: Emmanuel Lecharny [mailto:elecharny@apache.org]
Sent: Wednesday, April 11, 2012 2:43 PM
To: users@directory.apache.org
Subject: RE: Plea for help with search performance

ok. i'll check that tomorrow. the brst you can do is to fill a JIRA wjere you cam attach files.
be aware that those infos will be public.
Le 11 avr. 2012 19:34, <Carlo.Accorsi@ibs-ag.com> a écrit :

> Emmanuel ok great.. I just responded to Kiran and realized the
> attachments don't come through.
>
> I create everything with ldif files, restarting in between them.
> I start with a clean install, star the default instance once, the
> import the first file.
>
> ## begin file 1##
> #version: 1
>
> dn:
> ads-partitionId=cpro,ou=partitions,ads-directoryServiceId=default,ou=c
> onfig
> objectclass: top
> objectClass: ads-base
> objectclass: ads-partition
> objectclass: ads-jdbmPartition
> ads-indexes: apacheRdn
> ads-indexes: apacheSubLevel
> ads-indexes: apachePresence
> ads-indexes: apacheOneLevel
> ads-indexes: apacheOneAlias
> ads-indexes: apacheSubAlias
> ads-indexes: apacheAlias
> ads-indexes: entryUUID
> ads-indexes: entryCSN
> ads-indexes: o
> ads-indexes: krb5PrincipalName
> ads-indexes: objectClass
> ads-indexes: ou
> ads-indexes: uid
> ads-indexes: employeeNumber
> ads-indexes: displayName
> ads-indexes: cn
> ads-indexes: mail
> ads-partitionSuffix: o=cpro
> ads-jdbmpartitionoptimizerenabled: TRUE
> ads-partitioncachesize: 50000
> ads-partitionsynconwrite: TRUE
> ads-partitionid: cpro
> ads-enabled: TRUE
>
> dn:
> ou=indexes,ads-partitionId=cpro,ou=partitions,ads-directoryServiceId=d
> efault,ou=config
> objectclass: organizationalUnit
> objectclass: top
> ou: indexes
>
> dn:
> ads-indexAttributeId=apacheRdn,ou=indexes,ads-partitionId=cpro,ou=part
> itions,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: apacheRdn
> ads-enabled: TRUE
> ads-indexcachesize: 5000
>
> dn:
> ads-indexAttributeId=apacheSubLevel,ou=indexes,ads-partitionId=cpro,ou
> =partitions,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: apacheSubLevel
> ads-enabled: TRUE
> ads-indexcachesize: 5000
>
> dn:
> ads-indexAttributeId=apachePresence,ou=indexes,ads-partitionId=cpro,ou
> =partitions,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: apachePresence
> ads-enabled: TRUE
> ads-indexcachesize: 100
>
> dn:
> ads-indexAttributeId=apacheOneLevel,ou=indexes,ads-partitionId=cpro,ou
> =partitions,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: apacheOneLevel
> ads-enabled: TRUE
> ads-indexcachesize: 100
>
> dn:
> ads-indexAttributeId=apacheOneAlias,ou=indexes,ads-partitionId=cpro,ou
> =partitions,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: apacheOneAlias
> ads-enabled: TRUE
> ads-indexcachesize: 5000
>
> dn:
> ads-indexAttributeId=apacheSubAlias,ou=indexes,ads-partitionId=cpro,ou
> =partitions,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: apacheSubAlias
> ads-enabled: TRUE
> ads-indexcachesize: 100
>
> dn:
> ads-indexAttributeId=apacheAlias,ou=indexes,ads-partitionId=cpro,ou=pa
> rtitions,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: apacheAlias
> ads-enabled: TRUE
> ads-indexcachesize: 100
>
> dn:
> ads-indexAttributeId=o,ou=indexes,ads-partitionId=cpro,ou=partitions,a
> ds-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: o
> ads-enabled: TRUE
> ads-indexcachesize: 100
>
> dn:
> ads-indexAttributeId=krb5PrincipalName,ou=indexes,ads-partitionId=cpro
> ,ou=partitions,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: krb5PrincipalName
> ads-enabled: TRUE
> ads-indexcachesize: 100
>
> dn:
> ads-indexAttributeId=objectClass,ou=indexes,ads-partitionId=cpro,ou=pa
> rtitions,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: objectClass
> ads-enabled: TRUE
> ads-indexcachesize: 50000
>
> dn:
> ads-indexAttributeId=entryUUID,ou=indexes,ads-partitionId=cpro,ou=part
> itions,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: entryUUID
> ads-enabled: TRUE
> ads-indexcachesize: 5000
>
> dn:
> ads-indexAttributeId=entryCSN,ou=indexes,ads-partitionId=cpro,ou=parti
> tions,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: entryCSN
> ads-enabled: TRUE
> ads-indexcachesize: 5000
>
> dn:
> ads-indexAttributeId=ou,ou=indexes,ads-partitionId=cpro,ou=partitions,
> ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: ou
> ads-enabled: TRUE
> ads-indexcachesize: 100
>
> dn:
> ads-indexAttributeId=uid,ou=indexes,ads-partitionId=cpro,ou=partitions
> ,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: uid
> ads-enabled: TRUE
> ads-indexcachesize: 50000
>
> dn:
> ads-indexAttributeId=employeeNumber,ou=indexes,ads-partitionId=cpro,ou
> =partitions,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: employeeNumber
> ads-enabled: TRUE
> ads-indexcachesize: 50000
>
> dn:
> ads-indexAttributeId=displayName,ou=indexes,ads-partitionId=cpro,ou=pa
> rtitions,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-jdbmIndex
> objectclass: ads-base
> objectclass: top
> ads-indexattributeid: displayName
> ads-enabled: TRUE
> ads-indexcachesize: 50000
>
> dn:
> ads-indexAttributeId=cn,ou=indexes,ads-partitionId=cpro,ou=partitions,
> ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-base
> objectclass: ads-jdbmIndex
> objectclass: top
> ads-indexattributeid: cn
> ads-enabled: TRUE
> ads-indexcachesize: 50000
>
> dn:
> ads-indexAttributeId=mail,ou=indexes,ads-partitionId=cpro,ou=partition
> s,ads-directoryServiceId=default,ou=config
> objectclass: ads-index
> objectclass: ads-base
> objectclass: ads-jdbmIndex
> objectclass: top
> ads-indexattributeid: mail
> ads-enabled: TRUE
> ads-indexcachesize: 50000
>
> #Change password policy to allow hashed passwords from ldif
> dn:
> ads-pwdId=default,ou=passwordPolicies,ads-interceptorId=authentication
> Interceptor,ou=interceptors,ads-directoryServiceId=default,ou=config
> changetype: modify
> replace: ads-pwdcheckquality
> ads-pwdcheckquality: 0
>
> ## end file 1 ###
>
> Then I restart the server and re connect
>
> ## begin file 2 ##
> #version: 1
>
> dn: o=cpro
> objectClass: organization
> objectClass: top
> o: cpro
>
> dn: ou=int,o=cpro
> objectClass: organizationalUnit
> objectClass: top
> ou: int
> description: Internal
>
> dn: ou=users,ou=int,o=cpro
> objectClass: organizationalUnit
> objectClass: top
> ou: users
> description: Internal Users
>
> dn: ou=groups,ou=int,o=cpro
> objectClass: organizationalUnit
> objectClass: top
> ou: groups
> description: Internal Groups
>
> dn: ou=ext,o=cpro
> objectClass: organizationalUnit
> objectClass: top
> ou: ext
> description: External
>
> dn: ou=users,ou=ext,o=cpro
> objectClass: organizationalUnit
> objectClass: top
> ou: users
> description: External Users
>
> #External Password policy
> dn:
> ads-pwdId=cproext,ou=passwordPolicies,ads-interceptorId=authentication
> Interceptor,ou=interceptors,ads-directoryServiceId=default,ou=config
> objectclass: top
> objectclass: ads-base
> objectclass: ads-passwordPolicy
> ads-pwdattribute: userPassword
> ads-pwdid: cproext
> ads-enabled: TRUE
> ads-pwdallowuserchange: TRUE
> ads-pwdcheckquality: 0
> ads-pwdexpirewarning: 600
> ads-pwdfailurecountinterval: 30
> ads-pwdgraceauthnlimit: 5
> ads-pwdgraceexpire: 0
> ads-pwdinhistory: 5
> ads-pwdlockout: TRUE
> ads-pwdlockoutduration: 0
> ads-pwdmaxage: 0
> ads-pwdmaxdelay: 0
> ads-pwdmaxfailure: 5
> ads-pwdmaxidle: 0
> ads-pwdmaxlength: 0
> ads-pwdminage: 0
> ads-pwdmindelay: 0
> ads-pwdminlength: 5
> ads-pwdmustchange: FALSE
> ads-pwdsafemodify: FALSE
>
> dn:
> ads-pwdId=cproint,ou=passwordPolicies,ads-interceptorId=authentication
> Interceptor,ou=interceptors,ads-directoryServiceId=default,ou=config
> objectclass: ads-passwordPolicy
> objectclass: ads-base
> objectclass: top
> ads-pwdattribute: userPassword
> ads-pwdid: cproint
> ads-enabled: TRUE
> ads-pwdallowuserchange: TRUE
> ads-pwdcheckquality: 0
> ads-pwdexpirewarning: 10
> ads-pwdfailurecountinterval: 30
> ads-pwdgraceauthnlimit: 5
> ads-pwdgraceexpire: 0
> ads-pwdinhistory: 5
> ads-pwdlockout: TRUE
> ads-pwdlockoutduration: 0
> ads-pwdmaxage: 60
> ads-pwdmaxdelay: 0
> ads-pwdmaxfailure: 5
> ads-pwdmaxidle: 0
> ads-pwdmaxlength: 0
> ads-pwdminage: 0
> ads-pwdmindelay: 0
> ads-pwdminlength: 10
> ads-pwdmustchange: FALSE
> ads-pwdsafemodify: FALSE
>
> ## end file 2 ##
>
> I then update the partition cache to 50000 and restart again , before
> importing any users.
> Thanks.
>
> Regards,
> Carlo Accorsi
>
>
>
> -----Original Message-----
> From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
> Sent: Wednesday, April 11, 2012 1:17 PM
> To: users@directory.apache.org
> Subject: Re: Plea for help with search performance
>
> Ok,
>
> after some investigation, I'm pretty sure that the indexes are not
> created. When I create the indexes, I get those results :
>
> - 80000 entries injected into the server, with uid=<number>
> - displayName is indexed, uid is not indexed
>
> - first search :
> connection.search("dc=example,dc=com", "(displayName=12345Awg-Rosli,
> Awg-Abd-Rahim SMDS-UIA/G/MMO52D)", SearchScope.SUBTREE, "*" );
>
> result :
> Delta search : 16 ms
> entry found :
>     dn[n]: uid=12345,dc=example,dc=com
>     objectclass: organizationalPerson
>     objectclass: person
>     objectclass: inetOrgPerson
>     objectclass: top
>     title: Snr Operations Technician (D)
>     uid: 12345
>     description: UI - S
>     businesscategory: Ops MDS (Malaysia) Sdn Bhd
>     departmentnumber: SMDS - UIA/G/MMO52D
>     employeenumber: A-A-R.Awg-Rosli
>     givenname: Awg-Abd-Rahim
>     cn: Awg-Rosli, Awg-Abd-Rahim SMDS-UIA/G/MMO52D
>     sn: Awg-Rosli
>     telephonenumber: 555-1212
>     mail: A-A-R.Awg-Rosli@acme.com
>     displayname: 12345Awg-Rosli, Awg-Abd-Rahim SMDS-UIA/G/MMO52D
>
> - Second search :
> connection.search("dc=example,dc=com", "(displayName=34567*)",
> SearchScope.SUBTREE, "*" );
>
> result :
> Delta search substring : 9 ms
> entry found :
>     dn[n]: uid=34567,dc=example,dc=com
>     objectclass: organizationalPerson
>     objectclass: person
>     objectclass: inetOrgPerson
>     objectclass: top
>     title: Snr Operations Technician (D)
>     uid: 34567
>     description: UI - S
>     businesscategory: Ops MDS (Malaysia) Sdn Bhd
>     departmentnumber: SMDS - UIA/G/MMO52D
>     employeenumber: A-A-R.Awg-Rosli
>     givenname: Awg-Abd-Rahim
>     cn: Awg-Rosli, Awg-Abd-Rahim SMDS-UIA/G/MMO52D
>     sn: Awg-Rosli
>     telephonenumber: 555-1212
>     mail: A-A-R.Awg-Rosli@acme.com
>     displayname: 34567Awg-Rosli, Awg-Abd-Rahim SMDS-UIA/G/MMO52D
>
> -third search :
> connection.search("dc=example,dc=com", "(uid=67890)",
> SearchScope.SUBTREE, "*" );
>
> result :
> Delta search no index : 38985 ms
>
> entry found :
>     dn[n]: uid=67890,dc=example,dc=com
>     objectclass: organizationalPerson
>     objectclass: person
>     objectclass: inetOrgPerson
>     objectclass: top
>     title: Snr Operations Technician (D)
>     uid: 67890
>     description: UI - S
>     businesscategory: Ops MDS (Malaysia) Sdn Bhd
>     departmentnumber: SMDS - UIA/G/MMO52D
>     employeenumber: A-A-R.Awg-Rosli
>     givenname: Awg-Abd-Rahim
>     cn: Awg-Rosli, Awg-Abd-Rahim SMDS-UIA/G/MMO52D
>     sn: Awg-Rosli
>     telephonenumber: 555-1212
>     mail: A-A-R.Awg-Rosli@acme.com
>     displayname: 67890Awg-Rosli, Awg-Abd-Rahim SMDS-UIA/G/MMO52D
>
>
> So if the index is properly set, the search takes milliseconds to
> complete (which is expected). Otherwise, it can take tens of seconds...
>
> Two possibilities :
> - you don't initialize the index correctly. Please provide your
> configuration and if you are using ADS embedded, please provide the
> code that you use to embed it
> - we have an issue in the way we initialize the index, which is more
> likely to be the problem, as I fixed one issue in this area 2 weeks
> ago (in
> M7-SNAPSHOT) and I just fixed another one while doing the experiment...
>
> I'd like to rule out the first hypothesis first, but in any case, I'll
> do some more check tonite or tomorrow.
>
>
> Many thanks for your patience.
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>

Mime
View raw message