camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claus Ibsen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CAMEL-10900) OutOfMemoryError with big file in ftp or sftp producer with charset assigned
Date Mon, 27 Feb 2017 08:13:45 GMT

    [ https://issues.apache.org/jira/browse/CAMEL-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15885332#comment-15885332
] 

Claus Ibsen commented on CAMEL-10900:
-------------------------------------

Dont use the charset option for big files as it must read the content into memory to be able
to convert to another format.

> OutOfMemoryError with big file in ftp or sftp producer with charset assigned
> ----------------------------------------------------------------------------
>
>                 Key: CAMEL-10900
>                 URL: https://issues.apache.org/jira/browse/CAMEL-10900
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-ftp, camel-sftp
>    Affects Versions: 2.17.5
>         Environment: Centos 7, Java 1.8
>            Reporter: Luca Quintarelli
>            Priority: Minor
>
> During production of sftp or ftp file with charset assigned I have a OutOfMemory for
a file of 500MB.
> I think the problem is at this row: 
> org.apache.camel.component.file.remote.SftpOperations.doStoreFile(SftpOperations.java:875)
> and
> org.apache.camel.component.file.remote.FtpOperations.doStoreFile(FtpOperations.java:573)
> the row is :
>  is = new ByteArrayInputStream(exchange.getIn().getMandatoryBody(String.class).getBytes(charset));
> Try to convert the file to a String.
> Full stack trace: 
> org.apache.camel.component.file.GenericFileOperationFailedException: Cannot store file:
out/outfile.txt
> 	at org.apache.camel.component.file.remote.SftpOperations.doStoreFile(SftpOperations.java:902)
> 	at org.apache.camel.component.file.remote.SftpOperations.storeFile(SftpOperations.java:817)
> 	at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:277)
> 	at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:165)
> 	at org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:58)
> 	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
> 	at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:173)
> 	at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:341)
> 	at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:168)
> 	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> 	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
> 	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
> 	at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:681)
> 	at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:609)
> 	at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:238)
> 	at org.apache.camel.processor.Splitter.process(Splitter.java:108)
> 	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> 	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
> 	at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
> 	at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
> 	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> 	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
> 	at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:117)
> 	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> 	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
> 	at org.apache.camel.processor.FilterProcessor.process(FilterProcessor.java:57)
> 	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> 	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
> 	at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
> 	at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
> 	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> 	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
> 	at org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:44)
> 	at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
> 	at org.apache.camel.processor.loadbalancer.LoadBalancerSupport.process(LoadBalancerSupport.java:97)
> 	at org.apache.camel.component.quartz2.CamelJob.execute(CamelJob.java:58)
> 	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> 	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: org.apache.camel.InvalidPayloadException: No body available of type: java.lang.String
but has value: /tmp/myfile.tmp of type: java.io.File on: myfile.tmp. Caused by: Error during
type conversion from type: java.lang.String to the required type: java.lang.String with value
[Body is file based: /tmp/myfile.tmp] due java.lang.OutOfMemoryError: Java heap space. Exchange[ID-localhost-32802-1487864818750-0-13967030].
Caused by: [org.apache.camel.TypeConversionException - Error during type conversion from type:
java.lang.String to the required type: java.lang.String with value [Body is file based: /tmp/myfile.tmp]
due java.lang.OutOfMemoryError: Java heap space]
> 	at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:107)
> 	at org.apache.camel.component.file.remote.SftpOperations.doStoreFile(SftpOperations.java:865)
> 	... 57 more
> Caused by: org.apache.camel.TypeConversionException: Error during type conversion from
type: java.lang.String to the required type: java.lang.String with value [Body is file based:
/tmp/myfile.tmp] due java.lang.OutOfMemoryError: Java heap space
> 	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.createTypeConversionException(BaseTypeConverterRegistry.java:629)
> 	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:190)
> 	at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:105)
> 	... 58 more
> Caused by: org.apache.camel.RuntimeCamelException: java.lang.OutOfMemoryError: Java heap
space
> 	at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1690)
> 	at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1285)
> 	at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59)
> 	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:306)
> 	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:181)
> 	... 59 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 	at java.util.Arrays.copyOf(Arrays.java:3332)
> 	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
> 	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
> 	at java.lang.StringBuilder.append(StringBuilder.java:190)
> 	at org.apache.camel.converter.IOConverter.toString(IOConverter.java:318)
> 	at org.apache.camel.converter.IOConverter.toString(IOConverter.java:261)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1281)
> 	at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59)
> 	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:306)
> 	at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:181)
> 	at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:105)
> 	at org.apache.camel.component.file.remote.SftpOperations.doStoreFile(SftpOperations.java:865)
> 	at org.apache.camel.component.file.remote.SftpOperations.storeFile(SftpOperations.java:817)
> 	at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:277)
> 	at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:165)
> 	at org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:58)
> 	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
> 	at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:173)
> 	at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:341)
> 	at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:168)
> 	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> 	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
> 	at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
> 	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
> 	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
> 	at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:681)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message