directory-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankaj Rathod <pankajratho...@gmail.com>
Subject Re: apacheDS JVM process crashes with OutOfMemory
Date Thu, 15 Nov 2018 19:48:58 GMT
Hi Emmanuel,

>> Question : would it be possible for you to share your load test, so that
I can reproduc it on my server to see what's going on ?

There is no specific test case. We are observing the issue with any search
filter.

Perform below steps 10 times in a second
       Step1: SimpleBind to SSL connection
       Step2: Search
       Step3: Unbind

After 3-4 hours run of load test, I see multiple number of NioSocketSession
in apacheDS heap dumps.

I have shared you java project here which performs the load test. It will
be really helpful if you can reproduce the issue on your apacheDS instance
(M23 version only).

Thanks and Regards,
Pankaj Rathod

On Tue, Oct 16, 2018 at 10:34 AM Emmanuel Lécharny <elecharny@gmail.com>
wrote:

> Hi Pankaj,
>
> Le 16/10/2018 à 16:56, Pankaj Rathod a écrit :
> > Hi Emmanuel,
> >
> > Thanks for analyzing the heapdumps.
> >
> > You have mentioned two possible scenarios for this accumulation of
> > NioSocketSession objects.
> >
> > 1. - the clients start a connection, and never close it.
> > After finishing the testrun, I had checked number of connections to port
> > using netstat command. There were 0 connections. So I don't think this
> can
> > be the cause.
> > I checked below workitem is mentioned in apacheDS jira. Is this going to
> be
> > included in any upoming apacheDS release?
> >
> http://mail-archives.apache.org/mod_mbox/directory-dev/201808.mbox/%3CJIRA.13144434.1520885847000.120166.1534807140215@Atlassian.JIRA%3E
>
> I don't think so.
>
> The probem with this approach is that it would eat a lot of CPU checking
> sessions every second, for a very little gain:
> - most of the sessions will always be properly closed: they will be
> opened, do some operation, and then closed. If there is some delay
> between the opening and the closing, we will check every second for
> their idle status, for nothing if the client is properly written
> - if you have tens of thousands sessions, that will litterally kill the
> server as every second, it will check each session for its status.
>
> I'd rather depend on the underlying OS to properly handle TCP connections.
>
> >
> > 2. - the clients start a connection, close it, but the server never ditch
> > it. In your test, can you tell me if your client is closing the
> connection,
> > and how ?
> > Our testrun client executes new threads repetitively where every thread
> > performs below operations before terminating the thread.
> >             execute bind
> >             perform search
> >             execute unbind
>
> Sounds fine.
>
> >         We are observing memory growth if we enable or disable the
> "execute
> > unbind" task for the threads before termination.
>
> Sounds like a bug.
>
> >
> > We did some investigation in MINA for our test run and observed that the
> > session objects present in memory for longer time have status "Closing".
> We
> > are expecting this could be because of below issue
> >         https://issues.apache.org/jira/browse/DIRMINA-1021
>
> Unlikely. This JIRA has been fixed in MINA 2.0.14, ApacheDS M24 uses
> MINA 2.0.16
>
> >
> > The fix for this issue seems included in MINA version which is bundled in
> > apacheDS M24. Going to apacheDS M24/M25 will be time consuming process
> for
> > us. Although, we will plan to upgrade in future.
>
> Switching to ApacheDS M25 should not take that much time: you just have
> to reload the database from scratch, as the format has changed.
>
> >
> > Emmanuel, could you comment on below queries related to MINA issues for
> > apacheDS? Your response will help us deciding to test the fix for
> > DIRMINA-1021. If you suggest then I will forward this mailthread on MINA
> > forum instead.
> >         1. Can heapmemory get full because of this issue? Or something in
> > MINA/apacheDS will remove these accumulated CLOSING state
> NioSocketSession
> > objects? Will the heapdumps show the exact same picture (top consumer
> > hierarchy) if memory crashes because of this issue? We noticed in our
> > heapdumps that top consumer hierarchy is almost similar to the hprof I
> > shared with you. We observed large number of NioSocketSession objects
> were
> > occupying more than 95% of memory.
>
> I don't think MINA is the culprit here. There is clearly some
> investigation to be done to see if we are correctly handling closing
> session in ApacheDS.
>
> >         2. The mailthread for DIRMINA-1021 mentions code changes as
> > "writebuffer method should call removeNow method when IOException
> occurs".
> > Could you please comment if this single line code change needed for this
> > fix? (If you suggest, then I will forward this question on MINA forum
> > instead.)
>
> I don't reming from the top of my head what was the exact fix was, it
> was 3 years ago, more or less :-)
>
>
> Question : would it be possible for you to share your load test, so that
> I can reproduc it on my server to see what's going on ?
>
> Thanks !
>
> --
> Emmanuel Lecharny
>
> Symas.com
> directory.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message