directory-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiran Ayyagari <kayyag...@apache.org>
Subject Re: Possible regression for substring search on indexes?
Date Tue, 16 Oct 2012 13:57:37 GMT
On Tue, Oct 16, 2012 at 7:24 PM,  <Carlo.Accorsi@ibs-ag.com> wrote:
> Here are my files.. no, doesn't appear that I have them. Do these look right ?
>
> 0.9.2342.19200300.100.1.1.db
> 0.9.2342.19200300.100.1.1.lg
> 0.9.2342.19200300.100.1.1-uid.txt
> 0.9.2342.19200300.100.1.3.db
> 0.9.2342.19200300.100.1.3.lg
> 0.9.2342.19200300.100.1.3-mail.txt
> 0.9.2342.19200300.100.1.6.db
> 0.9.2342.19200300.100.1.6.lg
> 0.9.2342.19200300.100.1.6-roomNumber.txt
> 1.3.6.1.4.1.18060.0.4.1.2.3.db
> 1.3.6.1.4.1.18060.0.4.1.2.3.lg
> 1.3.6.1.4.1.18060.0.4.1.2.3-apachePresence.txt
> 1.3.6.1.4.1.18060.0.4.1.2.5.db
> 1.3.6.1.4.1.18060.0.4.1.2.5.lg
> 1.3.6.1.4.1.18060.0.4.1.2.50.db
> 1.3.6.1.4.1.18060.0.4.1.2.50-apacheRdn.txt
> 1.3.6.1.4.1.18060.0.4.1.2.5-apacheOneAlias.txt
> 1.3.6.1.4.1.18060.0.4.1.2.6.db
> 1.3.6.1.4.1.18060.0.4.1.2.6.lg
> 1.3.6.1.4.1.18060.0.4.1.2.6-apacheSubAlias.txt
> 1.3.6.1.4.1.18060.0.4.1.2.7.db
> 1.3.6.1.4.1.18060.0.4.1.2.7.lg
> 1.3.6.1.4.1.18060.0.4.1.2.7-apacheAlias.txt
> 1.3.6.1.4.1.42.2.27.8.1.23.db
> 1.3.6.1.4.1.42.2.27.8.1.23.lg
> 1.3.6.1.4.1.42.2.27.8.1.23-pwdPolicySubentry.txt
> 1.3.6.1.4.1.4203.666.1.7.db
> 1.3.6.1.4.1.4203.666.1.7.lg
> 1.3.6.1.4.1.4203.666.1.7-entryCSN.txt
> 1.3.6.1.4.1.5322.10.1.1.db
> 1.3.6.1.4.1.5322.10.1.1.lg
> 1.3.6.1.4.1.5322.10.1.1-krb5PrincipalName.txt
> 2.16.840.1.113730.3.1.241.db
> 2.16.840.1.113730.3.1.241.lg
> 2.16.840.1.113730.3.1.241-displayName.txt
> 2.16.840.1.113730.3.1.3.db
> 2.16.840.1.113730.3.1.3.lg
> 2.16.840.1.113730.3.1.3-employeeNumber.txt
> 2.5.18.5.db
> 2.5.18.5.lg
> 2.5.18.5-administrativeRole.txt
> 2.5.4.0.db
> 2.5.4.0.lg
> 2.5.4.0-objectClass.txt
> 2.5.4.10.db
> 2.5.4.10.lg
> 2.5.4.10-o.txt
> 2.5.4.11.db
> 2.5.4.11.lg
> 2.5.4.11-ou.txt
> 2.5.4.13.db
> 2.5.4.13.lg
> 2.5.4.13-description.txt
> 2.5.4.3.db
> 2.5.4.3.lg
> 2.5.4.31.db
> 2.5.4.31.lg
> 2.5.4.31-member.txt
> 2.5.4.3-cn.txt
> 2.5.4.4.db
> 2.5.4.4.lg
> 2.5.4.42.db
> 2.5.4.42.lg
> 2.5.4.42-givenName.txt
> 2.5.4.4-sn.txt
> master.db
>
looks fine
> Out of memory when searching for sn=Thomps*
>
ahhh, what is the size of heap memory allocated to JVM?

> jvm 1    | Server daemon died!
> jvm 1    | java.lang.OutOfMemoryError: Java heap space
> jvm 1    |      at java.nio.CharBuffer.allocate(CharBuffer.java:312)
> jvm 1    |      at java.nio.charset.CharsetEncoder.isLegalReplacement(CharsetEncoder.java:319)
> jvm 1    |      at java.nio.charset.CharsetEncoder.replaceWith(CharsetEncoder.java:267)
> jvm 1    |      at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:186)
> jvm 1    |      at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:209)
> jvm 1    |      at sun.nio.cs.SingleByteEncoder.<init>(SingleByteEncoder.java:39)
> jvm 1    |      at sun.nio.cs.MS1252$Encoder.<init>(MS1252.java:115)
> jvm 1    |      at sun.nio.cs.MS1252.newEncoder(MS1252.java:43)
> jvm 1    |      at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:215)
> jvm 1    |      at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:207)
> jvm 1    |      at java.lang.StringCoding.encode(StringCoding.java:266)
> jvm 1    |      at java.lang.StringCoding.encode(StringCoding.java:284)
> jvm 1    |      at java.lang.String.getBytes(String.java:986)
> jvm 1    |      at org.tanukisoftware.wrapper.WrapperManager.sendCommand(WrapperManager.java:3685)
> jvm 1    |      at org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:3804)
> jvm 1    |      at org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084)
> jvm 1    |      at java.lang.Thread.run(Thread.java:662)
> jvm 1    | [09:31:26] WARN [org.apache.directory.server.ldap.LdapProtocolHandler] - Unexpected
exception forcing session to close: sending disconnect notice to client.
> jvm 1    | java.lang.OutOfMemoryError: Java heap space
> jvm 1    |      at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232)
> jvm 1    |      at java.lang.StringCoding.encode(StringCoding.java:272)
> jvm 1    |      at java.lang.String.getBytes(String.java:946)
> jvm 1    |      at org.apache.directory.shared.util.Strings.getBytesUtf8(Strings.java:1587)
> jvm 1    |      at org.apache.directory.shared.ldap.model.entry.StringValue.readExternal(StringValue.java:448)
> jvm 1    |      at org.apache.directory.shared.ldap.model.entry.DefaultAttribute.readExternal(DefaultAttribute.java:2064)
> jvm 1    |      at org.apache.directory.server.core.partition.impl.btree.jdbm.EntrySerializer.deserialize(EntrySerializer.java:219)
> jvm 1    |      at jdbm.btree.BPage.deserialize(BPage.java:1255)
> jvm 1    |      at jdbm.btree.BPage.deserialize(BPage.java:84)
> jvm 1    |      at jdbm.recman.BaseRecordManager.fetch(BaseRecordManager.java:451)
> jvm 1    |      at jdbm.recman.CacheRecordManager.fetch(CacheRecordManager.java:264)
> jvm 1    |      at jdbm.btree.BPage.loadBPage(BPage.java:1017)
> jvm 1    |      at jdbm.btree.BPage.find(BPage.java:315)
> jvm 1    |      at jdbm.btree.BTree.browse(BTree.java:706)
> jvm 1    |      at jdbm.btree.BTree.find(BTree.java:548)
> jvm 1    |      at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.get(JdbmTable.java:312)
> jvm 1    |      at org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.lookup(AbstractBTreePartition.java:1183)
> jvm 1    |      at org.apache.directory.server.xdbm.search.evaluator.SubstringEvaluator.evaluate(SubstringEvaluator.java:135)
> jvm 1    |      at org.apache.directory.server.xdbm.search.evaluator.AndEvaluator.evaluate(AndEvaluator.java:109)
> jvm 1    |      at org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:146)
> jvm 1    |      at org.apache.directory.server.core.partition.impl.btree.EntryCursorAdaptor.get(EntryCursorAdaptor.java:40)
> jvm 1    |      at org.apache.directory.server.core.api.filtering.BaseEntryFilteringCursor.next(BaseEntryFilteringCursor.java:475)
> jvm 1    |      at org.apache.directory.server.ldap.handlers.SearchHandler.readResults(SearchHandler.java:377)
> jvm 1    |      at org.apache.directory.server.ldap.handlers.SearchHandler.doSimpleSearch(SearchHandler.java:830)
> jvm 1    |      at org.apache.directory.server.ldap.handlers.SearchHandler.handleIgnoringReferrals(SearchHandler.java:1137)
> jvm 1    |      at org.apache.directory.server.ldap.handlers.SearchHandler.handleWithReferrals(SearchHandler.java:1220)
> jvm 1    |      at org.apache.directory.server.ldap.handlers.SearchHandler.handle(SearchHandler.java:227)
> jvm 1    |      at org.apache.directory.server.ldap.handlers.SearchHandler.handle(SearchHandler.java:87)
> jvm 1    |      at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:209)
> jvm 1    |      at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:56)
> jvm 1    |      at org.apache.mina.handler.demux.DemuxingIoHandler.messageReceived(DemuxingIoHandler.java:232)
> jvm 1    |      at org.apache.directory.server.ldap.LdapProtocolHandler.messageReceived(LdapProtocolHandler.java:209)
>
>
>
>
> Regards,
> Carlo Accorsi
>
>
>
> -----Original Message-----
> From: ayyagarikiran@gmail.com [mailto:ayyagarikiran@gmail.com] On Behalf Of Kiran Ayyagari
> Sent: Tuesday, October 16, 2012 9:46 AM
> To: users@directory.apache.org
> Subject: Re: Possible regression for substring search on indexes?
>
> On Tue, Oct 16, 2012 at 6:35 PM,  <Carlo.Accorsi@ibs-ag.com> wrote:
>> OK thanks for the quick response. I've built and tested (revision
>> 1398757) but is still taking several minutes to return substring search. I'm doing
my searching via Studio and have exclusive access To the directory. Interestingly, it somehow
knows that count right away but doesn't return anything for several mintues.
>
>>
>> I just swapped out the jar (with the one from a week ago) in this case bc I was hoping
to validate the test without needing to rebuild and re-import all the users.
>> I would rebuild everything before for production but to test this correction, do
I need to do this?
>>
> better to rebuild the server and overwrite the existing jars
>> Also, months ago we had an explicit index on entryUUID but we removed it when we
didn't see it in the stock config.ldif.
>> entryCSN does have an index.
>>
> it is not needed anymore, cause we now use entryUUID as the *ID* of entry and master
table contains these IDs as keys
>> These are the indexed attrs on the partition. After the ldif import they all had
grown in size (on disk) so I'm fairly confident they're being populated.
>> Are we missing any that would work in conjunction with index searches? Thanks!
>>
>> apacheRdn
>> apacheSubLevel
>> apachePresence
>> apacheOneLevel
>> apacheOneAlias
>> apacheSubAlias
>> apacheAlias
>> entryCSN
>> o
>> krb5PrincipalName
>> objectClass
>> ou
>> uid
>> employeeNumber
>> displayName
>> cn
>> mail
>> roomNumber
>> pwdPolicySubEntry
>> member
>> description
>> givenName
>> sn
>> administrativeRole
>>
>>
> no, nothing is missing, otoh, do you really have the .db files related to ONE and SUB
level indices (not ONE and SUB level *aliases*) curious cause they are no longer created/used
by the server
>> Regards,
>> Carlo Accorsi
>>
>> -----Original Message-----
>> From: Emmanuel Lécharny [mailto:elecharny@gmail.com]
>> Sent: Tuesday, October 16, 2012 7:33 AM
>> To: users@directory.apache.org
>> Subject: Re: Possible regression for substring search on indexes?
>>
>> Le 10/15/12 11:52 PM, Carlo.Accorsi@ibs-ag.com a écrit :
>>> Hi, we're using M9 built from the trunk on Wednesday of last week. We have our
db setup as it has been for the past several months. There are 80k users.
>>>
>>> (sn=Thompson)  returns in miliseconds
>>> (sn=Thompso*)  returns after 3-4 minutes.
>>>
>>> This seems very much like an issue we had back and in June (and was fixed). 
Any help is most appreciated. We need this build for all the good work Karin did on the password
policy.
>>>
>>> Thank you, Carlo Accorsi
>>>
>> It should be fixed with
>> http://svn.apache.org/viewvc?rev=1398737&view=rev
>>
>> Can you give it a try ?
>>
>> --
>> Regards,
>> Cordialement,
>> Emmanuel Lécharny
>> www.iktek.com
>>
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com



-- 
Kiran Ayyagari
http://keydap.com

Mime
View raw message