camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johan Haleby <johan.hal...@gmail.com>
Subject Re: SocketSessionImpl in Mina component retained in memory indefinitely
Date Thu, 18 Feb 2010 09:35:19 GMT

Done!

Thanks a lot for your help so far.

/Johan



Claus Ibsen-2 wrote:
> 
> On Thu, Feb 18, 2010 at 10:02 AM, Johan Haleby <johan.haleby@gmail.com>
> wrote:
>>
>> I've now created a small Maven example project that demonstrate the
>> issue.
>> http://old.nabble.com/file/p27635967/camel-mina-outofmemory-example.zip
>> camel-mina-outofmemory-example.zip
>>
>> You should run the test in "MinaProducerRetainedInMemoryExample". To
>> speed
>> up the process I use the following VM arguments:
>> -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=25m -Xmx25m
>>
>> Here is a screen dump from VisualVM where you can see the aggregated
>> memory
>> consumption and eventually the high GC activity when memory is running
>> out:
>> http://old.nabble.com/file/p27635967/mina_camel_leak_visualvm.png
>>
>> Here's the screen dump from Eclipse MAT:
>> http://old.nabble.com/file/p27635967/mat_problem_suspects.png
>>
>> If you like I could also provide the heap dump. If there's anything else
>> I
>> can do to assist just tell me.
>>
>> /Johan
>>
> 
> Thanks Johan
> 
> Can you attach the sample to the JIRA ticket I just created
> https://issues.apache.org/activemq/browse/CAMEL-2484
> 
> If you grant the Apache license when attaching the project we can use
> that to create unit tests etc. in camel-mina to help solve the issue
> 
>>
>> Johan Haleby wrote:
>>>
>>> I'll try to find time for this tomorrow.
>>>
>>> /Johan
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Wed, Feb 17, 2010 at 4:59 PM, Johan Haleby <johan.haleby@gmail.com>
>>>> wrote:
>>>>>
>>>>> I don't suppose it matters much but there are actually 5 different
>>>>> Mina
>>>>> endpoints involved (not 4 as I mentioned earlier). These are:
>>>>> mina:tcp://localhost:6200?sync=false&textline=false
>>>>> mina:tcp://localhost:6201?sync=false&textline=false
>>>>> mina:tcp://localhost:6202?sync=false&textline=false
>>>>> mina:tcp://localhost:6203?sync=false&textline=false
>>>>> mina:tcp://localhost:6204?sync=false&textline=false
>>>>>
>>>>> It's always to one of these addresses to which a message is sent.
>>>>>
>>>>> I've also tried to do a producer.stop() after each
>>>>> producer.sendBody(..)
>>>>> but
>>>>> it doesn't help at all. I still get the OutOfMemoryError and the same
>>>>> number
>>>>> of MinaProducers are still retained in memory.
>>>>>
>>>>
>>>> I wonder what keeps it in the memory then. Could you create a small
>>>> project and attach it as a ZIP to a JIRA ticket?
>>>>
>>>>
>>>>
>>>>
>>>>> /Johan
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> On Wed, Feb 17, 2010 at 4:39 PM, Johan Haleby
>>>>>> <johan.haleby@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> In the test I'm sending messages between 4 different endpoints
but
>>>>>>> they're
>>>>>>> treated as dynamic I suppose. What I mean is that the message
body
>>>>>>> of
>>>>>>> each
>>>>>>> incoming message contains the reply address(es) (which could
be
>>>>>>> completely
>>>>>>> dynamic). Thus I use a recipientList to distribute a reply message
>>>>>>> to
>>>>>>> each
>>>>>>> reply address stated in the message body.
>>>>>>>
>>>>>>
>>>>>> I am sure if you close the producer template after use you should
>>>>>> probably not see this as it contains an internal cache.
>>>>>> But that one should only hit 1000, so I wonder why it grows to
>>>>>> 20000+.
>>>>>>
>>>>>> Well could you post a few samples of the endpoints you send to the
>>>>>> recipient list. If they are all 100% different then I would assume
>>>>>> Camel creates a new producer per message, as it would be a cache
miss
>>>>>> in the producer template.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> /Johan
>>>>>>>
>>>>>>>
>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> On Wed, Feb 17, 2010 at 4:17 PM, Johan Haleby
>>>>>>>> <johan.haleby@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> The debug message ("Closing session when complete at
address") is
>>>>>>>>> printed
>>>>>>>>> all
>>>>>>>>> the time so it should close the session. I've investigated
a bit
>>>>>>>>> further
>>>>>>>>> and
>>>>>>>>> I see that the problem seem to be that DefaultProducerTemplate
>>>>>>>>> holds
>>>>>>>>> a
>>>>>>>>> SpringCamelContext which in turn holds an ArrayList which
seem to
>>>>>>>>> contain
>>>>>>>>> all MinaProducers that has ever been created.
>>>>>>>>>
>>>>>>>>> I've uploaded a screenshot of the heap dump:
>>>>>>>>> http://old.nabble.com/file/p27625230/mina_camel_retaining_resources.png
>>>>>>>>> mina_camel_retaining_resources.png
>>>>>>>>>
>>>>>>>>
>>>>>>>> Are you using a lot of dynamic endpoints which are different?
>>>>>>>>
>>>>>>>>
>>>>>>>>> /Johan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> Could you google that exception and see what other
Mina users
>>>>>>>>>> have
>>>>>>>>>> encountered.
>>>>>>>>>>
>>>>>>>>>> And you see this DEBUG logging when you send to that
remote
>>>>>>>>>> endpoint?
>>>>>>>>>>
>>>>>>>>>>             if (LOG.isDebugEnabled()) {
>>>>>>>>>>                 LOG.debug("Closing session
when complete at
>>>>>>>>>> address: "
>>>>>>>>>> + endpoint.getAddress());
>>>>>>>>>>             }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Feb 17, 2010 at 3:27 PM, Johan Haleby
>>>>>>>>>> <johan.haleby@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I'm running into problems when using Camel (2.1
and 2.2) with
>>>>>>>>>>> Mina
>>>>>>>>>>> endpoints
>>>>>>>>>>> configured to disconnect the session after each
received
>>>>>>>>>>> message.
>>>>>>>>>>> When
>>>>>>>>>>> lots
>>>>>>>>>>> of messages are sent over a longer period of
time I eventually
>>>>>>>>>>> run
>>>>>>>>>>> out
>>>>>>>>>>> of
>>>>>>>>>>> memory and the system either throws OutOfMemoryError
or spends
>>>>>>>>>>> long
>>>>>>>>>>> intervals in GC. When analyzing the heap dump
in Eclipse MAT it
>>>>>>>>>>> finds
>>>>>>>>>>> this
>>>>>>>>>>> leak suspect:
>>>>>>>>>>>
>>>>>>>>>>> 27,840 instances of
>>>>>>>>>>> "org.apache.mina.transport.socket.nio.SocketSessionImpl",
loaded
>>>>>>>>>>> by
>>>>>>>>>>> "sun.misc.Launcher$AppClassLoader @ 0xad65d850"
occupy
>>>>>>>>>>> 20,662,464
>>>>>>>>>>> (20.42%)
>>>>>>>>>>> bytes
>>>>>>>>>>>
>>>>>>>>>>> The endpoint is configured like:
>>>>>>>>>>> mina:tcp://localhost:6200?sync=false&textline=false
and the
>>>>>>>>>>> message
>>>>>>>>>>> is
>>>>>>>>>>> sent
>>>>>>>>>>> to the endpoint using a
>>>>>>>>>>>  http://camel.apache.org/recipient-list.html
>>>>>>>>>>> recipient list .
>>>>>>>>>>>
>>>>>>>>>>> What could be the cause of this? E.g. do I need
to stop the
>>>>>>>>>>> producer
>>>>>>>>>>> template after each sent message? Right now the
ProducerTemplate
>>>>>>>>>>> is a
>>>>>>>>>>> singleton used concurrently by multiple threads.
>>>>>>>>>>>
>>>>>>>>>>> /Johan
>>>>>>>>>>> --
>>>>>>>>>>> View this message in context:
>>>>>>>>>>> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27624487.html
>>>>>>>>>>> Sent from the Camel - Users mailing list archive
at Nabble.com.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Claus Ibsen
>>>>>>>>>> Apache Camel Committer
>>>>>>>>>>
>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27625230.html
>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Claus Ibsen
>>>>>>>> Apache Camel Committer
>>>>>>>>
>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27625648.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27625979.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27635967.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
> 
> 
> 
> -- 
> Claus Ibsen
> Apache Camel Committer
> 
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
> 
> 

-- 
View this message in context: http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27636292.html
Sent from the Camel - Users mailing list archive at Nabble.com.


Mime
View raw message