Mailing-List: contact users-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@camel.apache.org
Received-SPF: pass (nike.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.in;
  h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=D50TaRtp/1aq4CnZHtZAfIv+ofZ4aW8PW3KEThn3Z2hz42XFWk/a9G8PpGaUXnDRnrgjHU5Ly6irbrquc9jTpUuAbL0Wywgb+lifBcwHZuPJbKrM5Ylsk2Ga7rAPvnLImp44wq+MlWWh/EIGq7hSSBhyAmO6y2AsBP/wCnswbQ8=;
References: <1392202216.87510.YahooMailNeo@web190704.mail.sg3.yahoo.com>
 <CAGB5yN=Rhua40kDwq7583cD0UZ0sD_pobgTSkFW9BwdzhWBi1Q@mail.gmail.com>
 <1392208624.13678.YahooMailNeo@web190703.mail.sg3.yahoo.com>
 <1392210119.96078.YahooMailNeo@web190702.mail.sg3.yahoo.com>
 <1392371562.3046.YahooMailNeo@web190705.mail.sg3.yahoo.com>
 <1392620411.38061.YahooMailNeo@web190704.mail.sg3.yahoo.com>
 <CAGB5yNk8i1U5ZARtAKm4+6gia4J9jDG2srov9GKt+xHCPsb_+Q@mail.gmail.com>
 <1392623394.97720.YahooMailNeo@web190703.mail.sg3.yahoo.com>
 <CAGB5yNkp35tSAyFE-nRYPG+iZmXdHoXARRKtAXo_OvfGoDC6Kg@mail.gmail.com>
Message-ID: <1392625414.3505.YahooMailNeo@web190706.mail.sg3.yahoo.com>
Date: Mon, 17 Feb 2014 16:23:34 +0800 (SGT)
From: Chirag Dewan <chirag.dewan22@yahoo.in>
Reply-To: Chirag Dewan <chirag.dewan22@yahoo.in>
Subject: Re: OOM issue due to MemoryIdempotentRepository
To: "users@camel.apache.org" <users@camel.apache.org>
In-Reply-To: 
 <CAGB5yNkp35tSAyFE-nRYPG+iZmXdHoXARRKtAXo_OvfGoDC6Kg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="-1489772017-415116080-1392625414=:3505"

---1489772017-415116080-1392625414=:3505
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Claus,=0A=0AI had this implementation before. With this I saw an unusual=
 behaviour. A thread was created for every file consumed by the template an=
d even though I called template.doneUoW the thread is not stopped. So if I =
call doneUoW(and not stop) the thread should be destroyed? And also on done=
UoW the inprogressRepository cache is cleared. Which is not getting cleared=
 at the moment(the reason I believe is causing the . Is there a possibility=
 that doneUoW is not working as it should?=0A=0AThanks .=0A=0AChirag=A0 =0A=
=0A=0A=0A=0A________________________________=0A From: Claus Ibsen <claus.ib=
sen@gmail.com>=0ATo: "users@camel.apache.org" <users@camel.apache.org>; Chi=
rag Dewan <chirag.dewan22@yahoo.in> =0ASent: Monday, 17 February 2014 1:28 =
PM=0ASubject: Re: OOM issue due to MemoryIdempotentRepository=0A =0A=0AOn M=
on, Feb 17, 2014 at 8:49 AM, Chirag Dewan <chirag.dewan22@yahoo.in> wrote:=
=0A> Hi Claus,=0A>=0A> So I dont need to start/stop the template for every =
file?=0A>=0A> Right now I create a consumerTemplate at satrtup and then on =
every method call(responsible for consuming file) I start the template,cons=
ume the file(template.receive) and stop the template.=0A>=0A> Is this the r=
ight approach?=0A>=0A=0ANo, only start|stop the template once on startup an=
d shutdown of your app.=0AAnd then call doneUoW for each file you consume.=
=0A=0A=0A> Thanks.=0A>=0A> Chirag=0A>=0A>=0A>=0A>=0A> _____________________=
___________=0A>=A0 From: Claus Ibsen <claus.ibsen@gmail.com>=0A> To: "users=
@camel.apache.org" <users@camel.apache.org>=0A> Sent: Monday, 17 February 2=
014 12:33 PM=0A> Subject: Re: OOM issue due to MemoryIdempotentRepository=
=0A>=0A>=0A> Hi=0A>=0A> You should likely reuse the consumer template inste=
ad. As this FAQ=0A> applies to it too=0A> http://camel.apache.org/why-does-=
camel-use-too-many-threads-with-producertemplate.html=0A>=0A> On Mon, Feb 1=
7, 2014 at 8:00 AM, Chirag Dewan <chirag.dewan22@yahoo.in> wrote:=0A>> Hi A=
ll,=0A>>=0A>> Any views on this?=0A>>=0A>> Actually my previous mail is not=
 understandable. Let me rephrase the issue.=0A>>=0A>> I am using consumer t=
emplate to consume single file using fileName property. I have a single ins=
tance of consumerTemplate and I call doneUoW and stop after consuming the f=
ile.=0A>>=0A>> I am running a test case where 10000 files are consumed by c=
onsumerTemplate(one file at a time) . After a while JVM goes OOM and crashe=
s. In heap dump,I can see large number of MemoryIdempotentRepository instan=
ces. I am not using idempotent=3Dtrue. I figured out that inProgressReposit=
ory also uses this cache.=0A>>=0A>> Is there something I am missing here?=
=0A>>=0A>> Thanks in advance.=0A>>=0A>> Chirag Dewan=0A>>=0A>>=0A>>=0A>>=0A=
>> ________________________________=0A>>=A0 From: Chirag Dewan <chirag.dewa=
n22@yahoo.in>=0A>> To: "users@camel.apache.org" <users@camel.apache.org>; C=
hirag Dewan <chirag.dewan22@yahoo.in>=0A>> Sent: Friday, 14 February 2014 3=
:22 PM=0A>> Subject: Re: OOM issue due to MemoryIdempotentRepository=0A>>=
=0A>>=0A>> Hi All,=0A>>=0A>> I have been looking around why a large number =
of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as =
well regarding inProgressRepository not getting cleared. Can that be the re=
ason here?=0A>>=0A>> Also,I am not using consumerTemplate to consume 1000o =
files. Does that also use inProgressRepository? And when is it cleared from=
 the memory?=0A>>=0A>> Any help is much appreciated.=0A>>=0A>> Thanks!=0A>>=
=0A>> Chirag Dewan=0A>>=0A>>=0A>>=0A>>=0A>> _______________________________=
_=0A>>=0A>> From: Chirag Dewan <chirag.dewan22@yahoo.in>=0A>> To: "users@ca=
mel.apache.org" <users@camel.apache.org>; Chirag Dewan <chirag.dewan22@yaho=
o.in>=0A>> Sent: Wednesday, 12 February 2014 6:31 PM=0A>> Subject: Re: OOM =
issue due to MemoryIdempotentRepository=0A>>=0A>>=0A>> Hi,=0A>>=0A>> And I =
am using readLock=3Dchanged. So that can be a reason too?=0A>>=0A>> Chirag =
Dewan=0A>>=0A>>=0A>>=0A>>=0A>> ________________________________=0A>>=0A>> F=
rom: Chirag Dewan <chirag.dewan22@yahoo.in>=0A>> To: "users@camel.apache.or=
g" <users@camel.apache.org>=0A>> Sent: Wednesday, 12 February 2014 6:07 PM=
=0A>> Subject: Re: OOM issue due to MemoryIdempotentRepository=0A>>=0A>>=0A=
>> Hi Claus,=0A>>=0A>> Thanks for the quick reply.=0A>>=0A>> It might well =
be. But there is one thing I would like to get some insight into.=0A>>=0A>>=
 If idempotent=3Dfalse(default) is used,will the LRU Cache still store the =
file referrence? Because that is what I can see in my memory dump.=0A>>=0A>=
> org.apache.camel.util.LRUCache=0A>>=A0  >org.apache.camel.com.googlecode.=
concurrentlinkedhashmap.ConcurrentLinkedHashMap=0A>>=0A>> I maybe doing som=
ething wrong here. Is there anything else I need to do to make sure that fi=
le is deleted after it has been processed and not stored in the cache?=0A>>=
=0A>> Thanks.=0A>>=0A>> Chirag Dewan=0A>>=0A>>=0A>>=0A>>=0A>>=0A>> ________=
________________________=0A>>=0A>> From: Claus Ibsen <claus.ibsen@gmail.com=
>=0A>> To: "users@camel.apache.org" <users@camel.apache.org>=0A>> Sent: Wed=
nesday, 12 February 2014 5:42 PM=0A>> Subject: Re: OOM issue due to MemoryI=
dempotentRepository=0A>>=0A>>=0A>> Hi=0A>>=0A>> Some days ago someone else =
posted about a OOME issue when using Storm.=0A>> It smells like a Storm + C=
amel issue somewhere.=0A>>=0A>>=0A>>=0A>> On Wed, Feb 12, 2014 at 11:50 AM,=
 Chirag Dewan <chirag.dewan22@yahoo.in> wrote:=0A>>> Hi All,=0A>>>=0A>>> I =
am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my =
application.=0A>>>=0A>>>=0A>>> Now I intent to delete the file after consum=
ing it. So I have delete=3Dtrue in my route. After consuming around 6000 fi=
les(out of 10000,as my test case) JVM goes Out of memory.=0A>>>=0A>>>=0A>>>=
 In my heap dump I can see the following trace as the major memory consumer=
s:=0A>>>=0A>>> java.util.ArrayList=0A>>>=A0 >java.lang.Object=0A>>>=0A>>>=
=A0 =A0  >org.apache.camel.management.DefaultManagementLifecycleStrategy$Pr=
eRegisterService=0A>>>=0A>>>=A0 =A0 =A0 =A0  >org.apache.camel.component.fi=
le.FileEndpoint=0A>>>=0A>>>=A0 =A0 =A0 =A0 =A0 =A0  >org.apache.camel.proce=
ssor.idempotent.MemoryIdempotentRepository=0A>>>=A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 =A0 >org.apache.camel.util.LRUCache=0A>>>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0  >org.apache.camel.com.googlecode.concurrentlinkedhashmap.Concu=
rrentLinkedHashMap=0A>>>=0A>>>=0A>>> So if I am not using indempotent,shoul=
d it still store the files in the memory? Is this memory repository used,ev=
en if I am deleting the files and not storing in memory? Is there a way I c=
an avoid that?=0A>>>=0A>>> I am testing this scenario under Storm as execut=
ion environment and with 2GB max heap space.=0A>>>=0A>>> Thanks in advance=
=0A>>>=0A>>> Chirag Dewan=0A>>=0A>>=0A>>=0A>> --=0A>> Claus Ibsen=0A>> ----=
-------------=0A>> Red Hat, Inc.=0A>> Email: cibsen@redhat.com=0A>> Twitter=
: davsclaus=0A>> Blog: http://davsclaus.com=0A>> Author of Camel in Action:=
 http://www.manning.com/ibsen=0A>> Make your Camel applications look hawt, =
try: http://hawt.io=0A=0A>=0A>=0A>=0A>=0A> --=0A> Claus Ibsen=0A> ---------=
--------=0A> Red Hat, Inc.=0A> Email: cibsen@redhat.com=0A> Twitter: davscl=
aus=0A> Blog: http://davsclaus.com=0A> Author of Camel in Action: http://ww=
w.manning.com/ibsen=0A> Make your Camel applications look hawt, try: http:/=
/hawt.io=0A=0A=0A=0A-- =0AClaus Ibsen=0A-----------------=0ARed Hat, Inc.=
=0AEmail: cibsen@redhat.com=0ATwitter: davsclaus=0ABlog: http://davsclaus.c=
om=0AAuthor of Camel in Action: http://www.manning.com/ibsen=0AMake your Ca=
mel applications look hawt, try: http://hawt.io
---1489772017-415116080-1392625414=:3505--