camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: OOM issue due to MemoryIdempotentRepository
Date Mon, 17 Feb 2014 07:03:22 GMT
Hi

You should likely reuse the consumer template instead. As this FAQ
applies to it too
http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html

On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <chirag.dewan22@yahoo.in> wrote:
> Hi All,
>
> Any views on this?
>
> Actually my previous mail is not understandable. Let me rephrase the issue.
>
> I am using consumer template to consume single file using fileName property. I have a
single instance of consumerTemplate and I call doneUoW and stop after consuming the file.
>
> I am running a test case where 10000 files are consumed by consumerTemplate(one file
at a time) . After a while JVM goes OOM and crashes. In heap dump,I can see large number of
MemoryIdempotentRepository instances. I am not using idempotent=true. I figured out that inProgressRepository
also uses this cache.
>
> Is there something I am missing here?
>
> Thanks in advance.
>
> Chirag Dewan
>
>
>
>
> ________________________________
>  From: Chirag Dewan <chirag.dewan22@yahoo.in>
> To: "users@camel.apache.org" <users@camel.apache.org>; Chirag Dewan <chirag.dewan22@yahoo.in>
> Sent: Friday, 14 February 2014 3:22 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi All,
>
> I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository.
I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the
reason here?
>
> Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository?
And when is it cleared from the memory?
>
> Any help is much appreciated.
>
> Thanks!
>
> Chirag Dewan
>
>
>
>
> ________________________________
>
> From: Chirag Dewan <chirag.dewan22@yahoo.in>
> To: "users@camel.apache.org" <users@camel.apache.org>; Chirag Dewan <chirag.dewan22@yahoo.in>
> Sent: Wednesday, 12 February 2014 6:31 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi,
>
> And I am using readLock=changed. So that can be a reason too?
>
> Chirag Dewan
>
>
>
>
> ________________________________
>
> From: Chirag Dewan <chirag.dewan22@yahoo.in>
> To: "users@camel.apache.org" <users@camel.apache.org>
> Sent: Wednesday, 12 February 2014 6:07 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi Claus,
>
> Thanks for the quick reply.
>
> It might well be. But there is one thing I would like to get some insight into.
>
> If idempotent=false(default) is used,will the LRU Cache still store the file referrence?
Because that is what I can see in my memory dump.
>
> org.apache.camel.util.LRUCache
>   >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>
> I maybe doing something wrong here. Is there anything else I need to do to make sure
that file is deleted after it has been processed and not stored in the cache?
>
> Thanks.
>
> Chirag Dewan
>
>
>
>
>
> ________________________________
>
> From: Claus Ibsen <claus.ibsen@gmail.com>
> To: "users@camel.apache.org" <users@camel.apache.org>
> Sent: Wednesday, 12 February 2014 5:42 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi
>
> Some days ago someone else posted about a OOME issue when using Storm.
> It smells like a Storm + Camel issue somewhere.
>
>
>
> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <chirag.dewan22@yahoo.in> wrote:
>> Hi All,
>>
>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>>
>>
>> Now I intent to delete the file after consuming it. So I have delete=true in my route.
After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>>
>>
>> In my heap dump I can see the following trace as the major memory consumers:
>>
>> java.util.ArrayList
>>  >java.lang.Object
>>
>>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>>
>>         >org.apache.camel.component.file.FileEndpoint
>>
>>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>>                  >org.apache.camel.util.LRUCache
>>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>
>>
>> So if I am not using indempotent,should it still store the files in the memory? Is
this memory repository used,even if I am deleting the files and not storing in memory? Is
there a way I can avoid that?
>>
>> I am testing this scenario under Storm as execution environment and with 2GB max
heap space.
>>
>> Thanks in advance
>>
>> Chirag Dewan
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io



-- 
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io

Mime
View raw message