camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Muhammad Rayhan <>
Subject SEDA processing throughput
Date Mon, 11 Apr 2011 11:17:10 GMT
Hi All

I have a case in which I need to stream 3 transaction log files in
real-time, process each line and do some filtering, transformation etc
and send the output to certain server. The current route I use:

    <camelContext id="camel" xmlns="">
    	<route id="route1">
    		<from uri="stream:file?fileName=/export/home/apps/smpp/sys/gw/tdr/MtTdr-1&amp;scanStream=true&amp;scanStreamDelay=50"/>
			<to uri="seda:input?timeout=0&amp;concurrentConsumers=16"/>
    	<route id="route2">
    		<from uri="stream:file?fileName=/export/home/apps/smpp/sys/gw2/tdr/MtTdr-1&amp;scanStream=true&amp;scanStreamDelay=50"/>
			<to uri="seda:input?timeout=0&amp;concurrentConsumers=16"/>
    	<route id="route3">
    		<from uri="stream:file?fileName=/export/home/apps/smpp/sys/gw3/tdr/MtTdr-1&amp;scanStream=true&amp;scanStreamDelay=50"/>
			<to uri="seda:input?timeout=0&amp;concurrentConsumers=16"/>
		<route id="route-x">
			<from uri="seda:input?timeout=0&amp;concurrentConsumers=16" />
			<log message="streaming TDR line = ${body}"/>
			<!-- Some filtering, transformation and aggregation here -->

The problem is, those 3 input files are produced by high throughput
application. And the SEDA consumer processing speed  is always lower,
so I ended up with late processing response time. For example, after
several minutes, let say at 16:15, the module is still processing the
log that is produced at 16:05. And the gap become bigger and bigger.

I try to remove the filtering, transforming part but still face the
same issue.I try to increase the concurrentConsumers but also no luck

Is there anyway to tune or improve the SEDA consumer throughput? Do I
miss something, or I design the route in incorrect way?

Twitter: @md_ray

View raw message