camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claus Ibsen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CAMEL-9735) camel-tarfile throws OutOfMemoryError when splitting large files
Date Tue, 22 Mar 2016 06:45:25 GMT

    [ https://issues.apache.org/jira/browse/CAMEL-9735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205896#comment-15205896
] 

Claus Ibsen commented on CAMEL-9735:
------------------------------------

Try test with the 2.17.0 RC at
http://camel.465427.n5.nabble.com/VOTE-Release-Apache-Camel-2-17-0-td5779375.html

> camel-tarfile throws OutOfMemoryError when splitting large files
> ----------------------------------------------------------------
>
>                 Key: CAMEL-9735
>                 URL: https://issues.apache.org/jira/browse/CAMEL-9735
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-tarfile
>    Affects Versions: 2.16.2
>            Reporter: Sverker Abrahamsson
>            Priority: Minor
>
> After the change in https://issues.apache.org/jira/browse/CAMEL-9375 I started to get
failures with the following stacktrace:
> {noformat}
> Caused by: java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(Arrays.java:3236)
>         at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
>         at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
>         at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
>         at org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:104)
>         at org.apache.camel.util.IOHelper.copy(IOHelper.java:199)
>         at org.apache.camel.util.IOHelper.copy(IOHelper.java:169)
>         at org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:218)
>         at org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:214)
>         at org.apache.camel.converter.stream.StreamCacheConverter.convertToStreamCache(StreamCacheConverter.java:82)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243)
>         at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59)
>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:293)
>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:168)
>         at org.apache.camel.dataformat.tarfile.TarIterator.createNextMessage(TarIterator.java:121)
>         at org.apache.camel.dataformat.tarfile.TarIterator.tryAdvanceToNext(TarIterator.java:100)
>         at org.apache.camel.dataformat.tarfile.TarIterator.hasNext(TarIterator.java:78)
>         at org.apache.camel.processor.Splitter$SplitterIterable$1.hasNext(Splitter.java:163)
>         at org.apache.camel.processor.Splitter.updateNewExchange(Splitter.java:256)
> {noformat}
> I.e. using StreamCache makes the whole entry go into memory and since they in my usecase
are several gigabytes it doesn't work very well. I realize that this might be caused since
I don't set any StreamCachingStrategy on my context but reading the corresponding doc part
I get the impression that the default settings should be that it will spool to disk after
reading 128 kbyte.
> However, I really don't want it to do that neither as that would 1) be a significant
overhead to write gigabytes of data to a temp file and 2) I might not even have that much
space where my temp directory is located. Instead the functionality that I want is that the
tarfile entries are streamed and written directly to the destination files without any unnecessary
caching in between.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message