camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jamesburn <James.B...@oup.com>
Subject Re: file trigger processing lots of files
Date Wed, 16 Mar 2016 13:06:00 GMT
Hi. Thanks for looking at this. Here are the routes below. 

I think the problem is confounded by my wanting to control processing the
messages asynchronously - we have to cope with a potential 100,000 files. If
I take away <threads> it works as I'd expect, with control-bus stopping the
route, but the messages appear to be processed one at a time. I think it's
perhaps using <threads> which I need to spend time on, but I would like to
know that I can use something like exchangeProperty.CamelBatchIndex to tell
me when they have all finished processing.

Thanks
James

<camelContext xmlns="http://camel.apache.org/schema/spring"
id="ONIXprocess">
	<propertyPlaceholder  id="props"
location="file:/opt/apache-servicemix-6.1.0/etc/constant.ONIXprocess.cfg"/>
	<route id="headerCollect">
		<from
uri="file:{{onix.header.pickup.location}}?delete=true&amp;include=.*header.*&amp;delay=1000"/>
		<log message="starting ProcessProducts route"/>
		<to uri="controlbus:route?routeId=processProducts&amp;action=start"/>
	</route>
	
	<route id="processProducts" autoStartup="false">
		<from
uri="file:{{onix.file.pickup.location}}?delete=true&amp;maxMessagesPerPoll=20&amp;include=.*xml&amp;sendEmptyMessageWhenIdle=true&amp;delay=500"/>
		<threads>
			
			<convertBodyTo type="String"/>
			
			<choice>
				<when>
					<xpath>/Product/ProductIdentifier/IDValue=9780193371644</xpath> 
					<log message="9780193371644 found in number
${exchangeProperty.CamelBatchIndex}"/>
					<delay>
						<constant>20000</constant>
					</delay>
					<log message="9780193371644 finished"/>
				</when>	
			</choice>
			<to uri="file:{{onix.file.dropoff.location}}"/>
			<log message="Processed record index no
${exchangeProperty.CamelBatchIndex} out of
${exchangeProperty.CamelBatchSize}"/>	
			
			<choice>
				<when>
					<simple>${exchangeProperty.CamelBatchComplete} == 'true'</simple>
					<log message="Batch of XML files finished?:
${exchangeProperty.CamelBatchComplete}"/>
					
						
					<to uri="controlbus:route?routeId=processProducts&amp;action=stop"/>
				</when>
			</choice>
		</threads>
	</route>
</camelContext>



--
View this message in context: http://camel.465427.n5.nabble.com/file-trigger-processing-lots-of-files-tp5778807p5779157.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Mime
View raw message