Mailing-List: contact user-help@synapse.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@synapse.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: Abid Khan-EXT <Abid.Khan@jeppesen.com>
To: "user@synapse.apache.org" <user@synapse.apache.org>
Date: Fri, 20 May 2011 10:13:42 -0600
Subject: RE: Please help: does Synapse leak memory?
Thread-Topic: Please help: does Synapse leak memory?
Thread-Index: AcwWrnrnaocAkxmhR5KgPphA43j4zwAWhFtw
Message-ID: 
 <83B2FA13648D5841982336487CE01FE01EB8BB18@DENPXCHMBX01.corp.gds.jeppesen.com>
References: 
 <83B2FA13648D5841982336487CE01FE01D6E4693@DENPXCHMBX01.corp.gds.jeppesen.com>
	<AANLkTi=ryKo9W27T1DCTWch_VcMkb1UKBJBN-51DuOaB@mail.gmail.com>
	<83B2FA13648D5841982336487CE01FE01D76A498@DENPXCHMBX01.corp.gds.jeppesen.com>
	<AANLkTik9FCCCpYynkGYm9yzSR2nX-vCiRSbUvATnwkcp@mail.gmail.com>
	<83B2FA13648D5841982336487CE01FE01D76A4D9@DENPXCHMBX01.corp.gds.jeppesen.com>
	<AANLkTi=4=P3xcziK18UhLVMjG8rGeFUy9xYbT-sKnLdV@mail.gmail.com>
	<83B2FA13648D5841982336487CE01FE01D76A630@DENPXCHMBX01.corp.gds.jeppesen.com>
	<AANLkTikzatsE+yezHjxuL-k5i1nU4ZEJB8zB8N=QBO1A@mail.gmail.com>
	<AANLkTiksmhXpaD06YQfQC+y=mA0FucpQ4h4N82oNyKQi@mail.gmail.com>
	<83B2FA13648D5841982336487CE01FE01DA21773@DENPXCHMBX01.corp.gds.jeppesen.com>
	<AANLkTikkNoPN3uOUtLjkoZe3wicahag5JF6uWH+q478r@mail.gmail.com>
	<83B2FA13648D5841982336487CE01FE01DA217B7@DENPXCHMBX01.corp.gds.jeppesen.com>
	<AANLkTin4-0ZAhDRT+iuQBuu2KdzoyFFwPpdfSY7KY87h@mail.gmail.com>
	<AANLkTimfP6p_z6FCru1PKc=ruAtHEX06xbEsqGcYfj+9@mail.gmail.com>
	<83B2FA13648D5841982336487CE01FE01DA217E8@DENPXCHMBX01.corp.gds.jeppesen.com>
	<AANLkTi=bmYO537TwA8NuM3E180wJY8iZ228T4KtTFWMg@mail.gmail.com>
	<83B2FA13648D5841982336487CE01FE01E4E8B87@DENPXCHMBX01.corp.gds.jeppesen.com>
	<83B2FA13648D5841982336487CE01FE01E8EE630@DENPXCHMBX01.corp.gds.jeppesen.com>
	<BANLkTimPhzAdJ30c-QcDgwaGGaenjsNjUQ@mail.gmail.com>
	<83B2FA13648D5841982336487CE01FE01E9EABEC@DENPXCHMBX01.corp.gds.jeppesen.com>
	<83B2FA13648D5841982336487CE01FE01EB485E0@DENPXCHMBX01.corp.gds.jeppesen.com>
	<BANLkTiksxQt9Zad9yze9p3PA2L3dn1rqWg@mail.gmail.com>
	<83B2FA13648D5841982336487CE01FE01EB48737@DENPXCHMBX01.corp.gds.jeppesen.com>
	<BANLkTinZ4okrazJS_TGjHs7uT4czbD47jA@mail.gmail.com>
	<83B2FA13648D5841982336487CE01FE01EB48879@DENPXCHMBX01.corp.gds.jeppesen.com>
 <BANLkTin9+s6pp0jUWGvrvkfCLEAoZU-awg@mail.gmail.com>
In-Reply-To: <BANLkTin9+s6pp0jUWGvrvkfCLEAoZU-awg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

The configuration that I have is huge, not only in number of mediators and =
sequences, but I am using lot of class mediators. That makes the configurat=
ion un useful for any who does not have the code.

What I will do is that I will try to reproduce it with synapse server and p=
ost the results here.

Thanks,
Abid


-----Original Message-----
From: Rajika Kumarasiri [mailto:rajika@wso2.com]
Sent: Thursday, May 19, 2011 11:22 PM
To: user@synapse.apache.org
Subject: Re: Please help: does Synapse leak memory?

Can you attach your Synapse configuration if possible? And tell which
endpoint has a typo ?

Rajika

On Thu, May 19, 2011 at 11:47 PM, Abid Khan-EXT <Abid.Khan@jeppesen.com>wro=
te:

> Thanks Hiranya,
>
> I have done some more research, and found out the it leaks memory only on
> the failures. Not on successful deliver.
>
> To give you the back ground, I had a typo in one of the endpoint URL. And=
 I
> had only two endpoints, all the message that were attempted to be deliver=
ed
> to that "wrong" url; failed. That is what lead to this leaking issue.
>
> On the other hand the server I was testing had -Xms1024m, and -Xmx2048m. =
So
> its little better than what I was thinking.
>
> I am still digging into and trying to find where the leak is. I will upda=
te
> if I found anything but so far it's a leak on the failures only.
>
> If you can find out if this is correct that Synapse could leak those two
> object on failures. That will help.
>
> Thanks,
> Abid
>
>
>
>
> -----Original Message-----
> From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
> Sent: Thursday, May 19, 2011 12:09 PM
> To: user@synapse.apache.org
> Subject: Re: Please help: does Synapse leak memory?
>
> Synapse creates an Axis2MessageContext for each message received. Each
> Axis2MessageContext instance encapsulates a MessageContext object. But
> Synapse does not keep them in memory for too long. As soon as messages ar=
e
> mediated and sent to the corresponding endpoints/clients, they are
> discarded. These objects could be fairly big since they contain message
> payload, properties and all other context information of messages. So if
> Synapse is keeping them in memory forever, we won't be able to run any
> scenario with Synapse for more than several hours. But as Paul mentioned =
we
> have seen some pretty heavy duty deployments of Synapse dealing with
> hundreds of transactions every second. One of the key strengths of Synaps=
e
> is that it can handle very high volumes of messages with a very small
> memory
> footprint.
>
> How much memory have you allocated for JBoss server? Also what is your
> Synapse configuration like? What's the memory usage pattern when you run
> Synapse standalone (ie without JBoss)?
>
> Thanks,
> Hiranya
>
> On Thu, May 19, 2011 at 9:17 PM, Abid Khan-EXT <Abid.Khan@jeppesen.com
> >wrote:
>
> > Hello Paul,
> >
> > What you are telling me is a good news. The facts I have here, are
> > different. I ran the server for a load-test and ran only 26000 message
> (of
> > about 1K each message size), that made the GC threads taking over every
> > thing, the java process was so stuck that the jboss-shutdown command
> could
> > not bring it down.
> >
> > And my trace log (memory profiling) showed that one instance of both
> > MessageContext and Axis2MessageContext were leaked for every message th=
at
> is
> > delivered/processed.
> >
> > I am using 1.2 GA of synapse. And I am running it in JBoss.
> >
> > And will be more than happy to send the thread dump, but that is very
> big.
> >
> > Once again thanks for getting back to me and I am looking for good
> pointers
> > from you very soon.
> >
> > Regards,
> > Abid
> >
> >
> >
> >
> > -----Original Message-----
> > From: Paul Fremantle [mailto:pzfreo@gmail.com]
> > Sent: Wednesday, May 18, 2011 8:39 PM
> > To: user@synapse.apache.org
> > Subject: Re: Please help: does Synapse leak memory?
> >
> > Abid
> >
> > Can you give us some idea of the flows? I know that Synapse is usually
> > memory safe - I have thread dumps from people who have pumped 134
> > terabytes of data, 1.4 billion messages and all this with a max heap
> > size of <2Gb and no leaks.
> >
> > Paul
> >
> > On Thu, May 19, 2011 at 1:49 AM, Abid Khan-EXT <Abid.Khan@jeppesen.com>
> > wrote:
> > > Hello all,
> > >
> > > I am troubleshooting an instance where I had used Synapse, the issue =
is
> > that after delivering few thousand message server runs out of memory.
> > >
> > > I investigated and found out that for each message that is delivered
> > Synapse is leaking following two objects:
> > >
> > > 1) org.apache.axis2.context.MessageContext
> > >
> > > 2) org.apache.synapse.core.axis2.Axis2MessageContext
> > >
> > > I thought that it might be related to threads and ThreadLocal, but wh=
en
> I
> > scanned the thread local of all 160+ threads those objects were not
> there.
> > >
> > > It seems that synapse is keeping those two object in its some interna=
l
> > object, that is difficult to find.
> > >
> > > Because of this problem those objects; server leaks memory at very hi=
gh
> > rate, and freezes after few hours of running?
> > >
> > > Any immediate response will be appreciated,
> > >
> > >
> > > Thanks,
> > > Abid
> > >
> >
> >
> >
> > --
> > Paul Fremantle
> > Co-Founder and CTO, WSO2
> > Apache Synapse PMC Chair
> > OASIS WS-RX TC Co-chair
> >
> > blog: http://pzf.fremantle.org
> > paul@wso2.com
> >
> > "Oxygenating the Web Service Platform", www.wso2.com
> >
>
>
>
> --
> Hiranya Jayathilaka
> Senior Software Engineer;
> WSO2 Inc.;  http://wso2.org
> E-mail: hiranya@wso2.com;  Mobile: +94 77 633 3491
> Blog: http://techfeast-hiranya.blogspot.com
>