Mailing-List: contact users-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@camel.apache.org
MIME-Version: 1.0
In-Reply-To: <1460732314764-5781179.post@n5.nabble.com>
References: <1459409155533-5780132.post@n5.nabble.com>
	<1459436310865-5780169.post@n5.nabble.com>
	<1460538377069-5780965.post@n5.nabble.com>
	<b74f28295f5740899faf3de809bc84f5@derote13de02.global.corp.sap>
	<1460550833912-5781003.post@n5.nabble.com>
	<8bbbcac692464c578bf6e06aff3a5280@derote13de02.global.corp.sap>
	<1460561153030-5781014.post@n5.nabble.com>
	<1460725145142-5781168.post@n5.nabble.com>
	<5710EBD5.4070600@j-b-s.de>
	<CAA8i-pZME51LmfADcSc2=r-+cw-VjeK-acUR9etU9eF85O30kw@mail.gmail.com>
	<1460732314764-5781179.post@n5.nabble.com>
Date: Fri, 15 Apr 2016 10:12:46 -0500
Message-ID: 
 <CAA8i-pafR-4OABka0PXSX3C=TDjPBAEPSy9K-UDF_35vZkXtQg@mail.gmail.com>
Subject: Re: Best Strategy to process a large number of rows in File
From: Brad Johnson <brad.johnson@mediadriver.com>
To: users@camel.apache.org
Content-Type: multipart/alternative; boundary=001a11c15716879d6005308771a9

--001a11c15716879d6005308771a9
Content-Type: text/plain; charset=UTF-8

I suspect the biggest problem you may be having here is that the file isn't
truly streaming but getting slurped into memory as a whole.  I don't know
that for certain but that test of two queues should show you that.

That might be throwaway code or it might be the basis for a next
implementation.  But it will at least let you see what you're dealing
with.  I wouldn't go too far past just reading/streaming the file in and
passing it on to the next queue so you can verify that you are getting
individual data records.

On Fri, Apr 15, 2016 at 9:58 AM, Michele <michele.mazzilli@finconsgroup.com>
wrote:

> How big is that object on the queue?
> As optimization, I changed generic LinkedHashMap in a bean SerialNumber
> that
> has this properties
>
> public class SerialNumber implements Serializable {
>
>         private static final long serialVersionUID = -9067015746823050409L;
>
>         private Date arg1;
>         private Date arg2;
>         private String arg3;
>         private String arg4;
>
>         // getter setter
> }
>
> Is it only a single line of data? Yes
>
> I try with seda:processAndStoreInQueue?concurrentConsumers=30 but the
> result
> is the same.
> I attached scrennshots with memory usage and live threads.
> memory-usage.png
> <http://camel.465427.n5.nabble.com/file/n5781179/memory-usage.png>
> live-threads.png
> <http://camel.465427.n5.nabble.com/file/n5781179/live-threads.png>
>
> I will try decompose the problem into a  number of separate seda:queues to
> verify performance and memory usage and let you know.
>
> Thanks a lot again for your support.
>
> Best Regards
>
> Michele
>
>
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Best-Strategy-to-process-a-large-number-of-rows-in-File-tp5779856p5781179.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

--001a11c15716879d6005308771a9--