camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michele <>
Subject Re: Best Strategy to process a large number of rows in File
Date Sat, 16 Apr 2016 13:04:49 GMT

as Ranx suggested, I tried with chain routes and attached screenshot of
memory usage  seda-memory-usage.png
<>  .
it is evident that there is a net improvement, on average 600 MB memory

The chain of routes configured is:

<route id="FileRetriever_Route">
	<to uri="seda:processAndStoreInQueue" />			
<route id="Splitter_Route">
	<from uri="seda:processAndStoreInQueue" />
	<unmarshal  ref="myBeanio" />

	<split streaming="true" executorServiceRef="threadPoolExecutor" >
				<to uri="seda:process.queue" />
				<log message="Message discarded ${in.header.CamelSplitIndex} - ${body}"
<route id="ProcessAndStoreInSedaQueue">
	<from uri="seda:process.queue?concurrentConsumers=10" />
	<process ref="BodyEnricherProcessor" />
	<marshal ref="Gson" />
	<log message="Store in SEDA Queue: ${in.header.CamelSplitIndex} - ${body}"
	<to uri="seda:readyToSendRS.queue" />

<route id="ProcessMessageData_Route" errorHandlerRef="DLQErrorHandler" >
	<from uri="seda:readyToSendRS.queue?concurrentConsumers=10" />
	<throttle timePeriodMillis="1000" asyncDelayed="true">
		<enrich uri="direct:crm-login" strategyRef="OAuthStrategy" />

		<unmarshal ref="Gson" />
				<simple>${in.header.Result} == 'ERROR'</simple>
				<log message="CRM Response: ${body}" loggingLevel="ERROR"/>

Store in SEDA Queue: 36913 -
and Throttling Policy in action:
 INFO  | d #46 - Throttle | CRMLogin_Route  ...

In addition, from my JUnit Test Camel is fastest (less than 1 minute) in the
processing  Read-Unmarshal-Split-Process-MockResult.

So, when I repalced seda:readyToSendRS.queue with
activemq:queue:readyToSendRS, the memory usage triples.

I can not use SEDA Component because in according to documentation:
this component does not implement any kind of persistence or recovery, if
the VM terminates while messages are yet to be processed. If you need
persistence, reliability or distributed SEDA, try using either JMS or

So, Can ActiveMQ manage a large number of messages in few seconds or
minutes? Or Do I need  to defined a load balancer on Broker? 

Thanks a lot for your support.

Best Regards


View this message in context:
Sent from the Camel - Users mailing list archive at

View raw message