synapse-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abid Khan-EXT <Abid.K...@jeppesen.com>
Subject RE: Please help: does Synapse leak memory?
Date Thu, 19 May 2011 18:17:20 GMT
Thanks Hiranya,

I have done some more research, and found out the it leaks memory only on the failures. Not
on successful deliver.

To give you the back ground, I had a typo in one of the endpoint URL. And I had only two endpoints,
all the message that were attempted to be delivered to that "wrong" url; failed. That is what
lead to this leaking issue.

On the other hand the server I was testing had -Xms1024m, and -Xmx2048m. So its little better
than what I was thinking.

I am still digging into and trying to find where the leak is. I will update if I found anything
but so far it's a leak on the failures only.

If you can find out if this is correct that Synapse could leak those two object on failures.
That will help.

Thanks,
Abid




-----Original Message-----
From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com] 
Sent: Thursday, May 19, 2011 12:09 PM
To: user@synapse.apache.org
Subject: Re: Please help: does Synapse leak memory?

Synapse creates an Axis2MessageContext for each message received. Each
Axis2MessageContext instance encapsulates a MessageContext object. But
Synapse does not keep them in memory for too long. As soon as messages are
mediated and sent to the corresponding endpoints/clients, they are
discarded. These objects could be fairly big since they contain message
payload, properties and all other context information of messages. So if
Synapse is keeping them in memory forever, we won't be able to run any
scenario with Synapse for more than several hours. But as Paul mentioned we
have seen some pretty heavy duty deployments of Synapse dealing with
hundreds of transactions every second. One of the key strengths of Synapse
is that it can handle very high volumes of messages with a very small memory
footprint.

How much memory have you allocated for JBoss server? Also what is your
Synapse configuration like? What's the memory usage pattern when you run
Synapse standalone (ie without JBoss)?

Thanks,
Hiranya

On Thu, May 19, 2011 at 9:17 PM, Abid Khan-EXT <Abid.Khan@jeppesen.com>wrote:

> Hello Paul,
>
> What you are telling me is a good news. The facts I have here, are
> different. I ran the server for a load-test and ran only 26000 message (of
> about 1K each message size), that made the GC threads taking over every
> thing, the java process was so stuck that the jboss-shutdown command could
> not bring it down.
>
> And my trace log (memory profiling) showed that one instance of both
> MessageContext and Axis2MessageContext were leaked for every message that is
> delivered/processed.
>
> I am using 1.2 GA of synapse. And I am running it in JBoss.
>
> And will be more than happy to send the thread dump, but that is very big.
>
> Once again thanks for getting back to me and I am looking for good pointers
> from you very soon.
>
> Regards,
> Abid
>
>
>
>
> -----Original Message-----
> From: Paul Fremantle [mailto:pzfreo@gmail.com]
> Sent: Wednesday, May 18, 2011 8:39 PM
> To: user@synapse.apache.org
> Subject: Re: Please help: does Synapse leak memory?
>
> Abid
>
> Can you give us some idea of the flows? I know that Synapse is usually
> memory safe - I have thread dumps from people who have pumped 134
> terabytes of data, 1.4 billion messages and all this with a max heap
> size of <2Gb and no leaks.
>
> Paul
>
> On Thu, May 19, 2011 at 1:49 AM, Abid Khan-EXT <Abid.Khan@jeppesen.com>
> wrote:
> > Hello all,
> >
> > I am troubleshooting an instance where I had used Synapse, the issue is
> that after delivering few thousand message server runs out of memory.
> >
> > I investigated and found out that for each message that is delivered
> Synapse is leaking following two objects:
> >
> > 1) org.apache.axis2.context.MessageContext
> >
> > 2) org.apache.synapse.core.axis2.Axis2MessageContext
> >
> > I thought that it might be related to threads and ThreadLocal, but when I
> scanned the thread local of all 160+ threads those objects were not there.
> >
> > It seems that synapse is keeping those two object in its some internal
> object, that is difficult to find.
> >
> > Because of this problem those objects; server leaks memory at very high
> rate, and freezes after few hours of running?
> >
> > Any immediate response will be appreciated,
> >
> >
> > Thanks,
> > Abid
> >
>
>
>
> --
> Paul Fremantle
> Co-Founder and CTO, WSO2
> Apache Synapse PMC Chair
> OASIS WS-RX TC Co-chair
>
> blog: http://pzf.fremantle.org
> paul@wso2.com
>
> "Oxygenating the Web Service Platform", www.wso2.com
>



-- 
Hiranya Jayathilaka
Senior Software Engineer;
WSO2 Inc.;  http://wso2.org
E-mail: hiranya@wso2.com;  Mobile: +94 77 633 3491
Blog: http://techfeast-hiranya.blogspot.com

Mime
View raw message