directory-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Karasulu <akaras...@gmail.com>
Subject Re: ApacheDS causing very high load
Date Thu, 10 Dec 2009 07:25:32 GMT
Whatever you guys are most comfortable with.  The idea here is to get a view
of what the call chain looks like and which methods are executing most of
the time.  This will give us an idea of what kind of infinite loop we're
dealing with which steals one or more threads and hikes up the CPU.

Thanks,
Alex

On Thu, Dec 10, 2009 at 12:59 AM, Ioannis Mavroukakis <
imavroukakis@gameaccount.com> wrote:

> Sure, I'll arrange this with our sysadm. Do you have anything particular in
> mind or will jvisualvm do?
>
> Y.
>
>
> On 10 Dec 2009, at 04:00, Alex Karasulu wrote:
>
>  It would be nice to run your server with a profiler to be able to catch
>> just
>> when it does get into this state.  This would be very valuable since we'd
>> see exactly where the code is getting stuck in this terrible loop that
>> eats
>> your CPU.
>>
>> You think you can give that a try Ioannis? If you can I'm sure we can work
>> together to nip this bug quickly.
>>
>> Thanks,
>> Alex
>>
>> On Wed, Dec 9, 2009 at 11:04 AM, Ioannis Mavroukakis <
>> imavroukakis@gameaccount.com> wrote:
>>
>>  Hi Emmanuel,
>>>
>>> Don't think it is too, as the same behaviour persists when we shut down
>>> ADS
>>> and re-start it. Only clearing the ADS directory allows it to recover! I
>>> have kept the directory contents though, is there
>>> any way I can run any forensics on them ?
>>>
>>> Thanks,
>>>
>>> Y.
>>>
>>> On 9 Dec 2009, at 15:21, Emmanuel LŽcharny wrote:
>>>
>>> Ioannis Mavroukakis a écrit :
>>>
>>>>
>>>>  Hey fellow listers.
>>>>>
>>>>>  Hi Ioannis,
>>>>
>>>> if you get 100% loaded CPU the it might be MINA which is the problem.
>>>> There is a painful bug in the way the JVM handle epoll, and in certain
>>>> conditions, it jump to 100% CPU and never goes down. It seems to be a
>>>> problem when a connection is opened, and closed before the selector can
>>>> see
>>>> the close : the select() methods returns at least one selectedKey, but
>>>> as
>>>> the connection has been closed, it does nothing but loop again (as you
>>>> can
>>>> imagine, the loop is infinite).
>>>>
>>>> There is a fix for that, but it's experimental, in MINA branch
>>>> http://svn.apache.org/repos/asf/mina/branches/select-fix/. I will
>>>> commit
>>>> this fix immediately in the MINA trunk, and you'll be able to bump up
>>>> the
>>>> MINA version to 2.0.0-RC2-SNAPSHOT in ADS.
>>>>
>>>> Not sure though that it's really your problem.
>>>>
>>>>
>>>>> I've got an issue with an embedded instance of apacheds in a trivial
>>>>> piece of java code. For reasons I have been unable to diagnose
>>>>> so far, it will run happily for about a week or so, then it will
>>>>> suddenly
>>>>> ramp up the load on the server it's running on, with the only
>>>>> way to recover it, being to delete the physical directory where the
>>>>> records are stored and start from scratch. What's puzzling me is
>>>>> that it's definitely not load related as the queries an operations to
>>>>> LDAP are very lightweight and infrequent. What I do get in the logs
>>>>> are loads of these
>>>>>
>>>>>
>>>>> pool-4-thread-18 ERROR registries.DefaultAttributeTypeRegistry -
>>>>> attributeType w/ OID 1.3.6.1.4.1.1466.115.121.1.12 not registered!
>>>>> pool-4-thread-19 ERROR registries.DefaultAttributeTypeRegistry -
>>>>> attributeType w/ OID 1.3.6.1.4.1.1466.115.121.1.12 not registered!
>>>>> pool-4-thread-19 WARN  context.SearchingOperationContext - Requested
>>>>> attribute dn does not exist in the schema, it will be ignored
>>>>> pool-4-thread-18 WARN  context.SearchingOperationContext - Requested
>>>>> attribute dn does not exist in the schema, it will be ignored
>>>>> NioProcessor-2 WARN  ldap.LdapProtocolHandler - Null LdapSession given
>>>>> to
>>>>> cleanUpSession.
>>>>> pool-4-thread-19 ERROR registries.DefaultAttributeTypeRegistry -
>>>>> attributeType w/ OID 1.3.6.1.4.1.1466.115.121.1.12 not registered!
>>>>> pool-4-thread-17 ERROR registries.DefaultAttributeTypeRegistry -
>>>>> attributeType w/ OID 1.3.6.1.4.1.1466.115.121.1.12 not registered!
>>>>>
>>>>>
>>>>> and
>>>>>
>>>>>
>>>>> pool-4-thread-19 WARN  ldap.LdapProtocolHandler - Unexpected exception
>>>>> forcing session to close: sending disconnect notice to client.
>>>>> java.lang.NullPointerException
>>>>>     at
>>>>>
>>>>> org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:129)
>>>>>     at
>>>>>
>>>>> org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:56)
>>>>>     at
>>>>>
>>>>> org.apache.mina.handler.demux.DemuxingIoHandler.messageReceived(DemuxingIoHandler.java:232)
>>>>>     at
>>>>>
>>>>> org.apache.directory.server.ldap.LdapProtocolHandler.messageReceived(LdapProtocolHandler.java:194)
>>>>>     at
>>>>>
>>>>> org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:721)
>>>>>     at
>>>>>
>>>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:433)
>>>>>     at
>>>>>
>>>>> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47)
>>>>>     at
>>>>>
>>>>> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:801)
>>>>>     at
>>>>>
>>>>> org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:71)
>>>>>     at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)
>>>>>     at
>>>>>
>>>>> org.apache.mina.filter.executor.UnorderedThreadPoolExecutor$Worker.runTask(UnorderedThreadPoolExecutor.java:480)
>>>>>     at
>>>>>
>>>>> org.apache.mina.filter.executor.UnorderedThreadPoolExecutor$Worker.run(UnorderedThreadPoolExecutor.java:434)
>>>>>     at java.lang.Thread.run(Thread.java:619)
>>>>> pool-4-thread-19 WARN  ldap.LdapProtocolHandler - Null LdapSession
>>>>> given
>>>>> to cleanUpSession.
>>>>> NioProcessor-3 WARN  ldap.LdapProtocolHandler - Null LdapSession given
>>>>> to
>>>>> cleanUpSession.
>>>>>
>>>>>
>>>>> The only thing I can think of causing the NPE's is a monitoring
>>>>> application that we have, that connects and disconnects from ADS to
>>>>> confirm
>>>>> that it's still there...Apart from that I have no plausible explanation
>>>>> for the high load. I'm currently using 1.5.6 compiled from source.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Yiannis
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ______________________________________________________________________
>>>> This email has been scanned by the MessageLabs Email Security System.
>>>> For more information please visit
>>>> http://www.messagelabs.com/email______________________________________________________________________
>>>>
>>>>
>>>
>>>
>>
>> --
>> Alex Karasulu
>> My Blog :: http://www.jroller.com/akarasulu/
>> Apache Directory Server :: http://directory.apache.org
>> Apache MINA :: http://mina.apache.org
>>
>> ______________________________________________________________________
>> This email has been scanned by the MessageLabs Email Security System.
>> For more information please visit http://www.messagelabs.com/email
>> ______________________________________________________________________
>>
>
>


-- 
Alex Karasulu
My Blog :: http://www.jroller.com/akarasulu/
Apache Directory Server :: http://directory.apache.org
Apache MINA :: http://mina.apache.org

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message