camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vcheruvu <v_cher...@hotmail.com>
Subject Java heap space issue with reading large CSV file
Date Wed, 18 Aug 2010 02:13:48 GMT

Hi,

I have 45MB CSV file. I have created camel route context which simply reads
from the csv file and routes it to bean for further processing. In doing so,
i have been getting Java heap space error while Camel tries to process 218K
lines. Any tips to get around this heap space error other than increasing
memory?  Is there way to read the file in certain sizes or any memory
optimizing techniques in Camel? 

Route config:

	<route
id="file://source/pats?delay=10000&delete=false&exclude=.done&move=.done/${file:name.noext}-${date:now:yyyyMMddHHmmss}&moveFailed=.error/${file:name.noext}-${date:now:yyyyMMddHHmmss}.error&autoCreate=false"/>
				<unmarshal> <csv/> </unmarshal>
			<split> 
				<simple>body</simple>
				<to uri="bean:saveContractData?method=persistContractData"/>
			</split>
			
		</route>


 JVM config = 
 -Xms1024m -Xmx1024m -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=8
-XX:NewSize=128m -XX:MaxNewSize=128m -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:PermSize=64m
-XX:MaxPermSize=64m

2010-08-18 11:57:53,240 INFO  [main] DefaultCamelContext.start:1025 - Apache
Camel 2.3.0 (CamelContext: camel) started in 735 millis
2010-08-18 11:58:58,453 FATAL [Camel Thread 1 -
file://source/pats?autoCreate=false&delay=10000&delete=false&exclude=.done&move=.done%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D&moveFailed=.error%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D.error]
ScheduledPollConsumer.run:121 - Consumer
Consumer[file://source/pats?autoCreate=false&delay=10000&delete=false&exclude=.done&move=.done%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D&moveFailed=.error%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D.error]
could not poll endpoint:
file://source/pats?autoCreate=false&delay=10000&delete=false&exclude=.done&move=.done%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D&moveFailed=.error%2F%24%7Bfile%3Aname.noext%7D-%24%7Bdate%3Anow%3AyyyyMMddHHmmss%7D.error
caused by: Java heap space
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2882)
	at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
	at java.lang.StringBuilder.append(StringBuilder.java:119)
	at java.lang.StringBuilder.append(StringBuilder.java:115)
	at java.util.AbstractCollection.toString(AbstractCollection.java:422)
	at java.lang.String.valueOf(String.java:2826)
	at java.lang.StringBuilder.append(StringBuilder.java:115)
	at
org.apache.camel.component.file.GenericFileMessage.toString(GenericFileMessage.java:58)
	at java.lang.String.valueOf(String.java:2826)
	at java.lang.StringBuilder.append(StringBuilder.java:115)
	at org.apache.camel.impl.DefaultExchange.toString(DefaultExchange.java:79)
	at java.lang.String.valueOf(String.java:2826)
	at java.lang.StringBuilder.append(StringBuilder.java:115)
	at
org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:74)
	at
org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:52)
	at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:170)
	at
org.apache.camel.processor.UnitOfWorkProcessor.done(UnitOfWorkProcessor.java:86)
	at
org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:75)
	at
org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
	at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
	at
org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:269)
	at
org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:137)
	at
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:103)
	at
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:98)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
	at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
	at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
	at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)




Kind regards,
-Vid-
-- 
View this message in context: http://camel.465427.n5.nabble.com/Java-heap-space-issue-with-reading-large-CSV-file-tp2638903p2638903.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Mime
View raw message