camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ganga_camel <>
Subject How to identify the last record that is getting processed?
Date Mon, 07 Dec 2015 15:28:44 GMT

I am trying to process a 10GB file (fixed width file) by performing the
below steps
1. Split the file per record based on the newline
			<from uri="file:src/data?noop=true" />
			<log message="Started Processing" loggingLevel="INFO" />
			<split streaming="true">
				<tokenize token="\n" />
				<to uri="seda:WriteToFile2" />
2. Perform Filter and transform the data that will result in 3 different CSV
files (| delimited)
3. Add header column to all the 3 files
4. Zip the end result into a GZip file

The problem I am facing is in the step 3 and 4
Below code is how I am trying to aggregate and write to a file, I am using
<process ref="AddHeader" /> to add header before I the data into the file,
however, the header gets added multiple times, one for every 30 seconds,
within the output file, as the aggregation completionInterval is set to
"30000". I am facing the same issue with creating the ZIP file.

Is there a way to identify the last record being processed and store the
flag in a global variable, using this variable value I can add the header
and zip the file once all the records are processed.

			<from uri="seda:NullProcessing?concurrentConsumers=1" />
			<aggregate strategyRef="aggregatorStrategy"
				<to uri="seda:processedRejects" />

			<from uri="seda:processedRejects" />
			<process ref="AddHeader" />
			<setHeader headerName="CamelFileName">
			<to uri="file:src/data/output?fileExist=Append" />

View this message in context:
Sent from the Camel - Users mailing list archive at

View raw message