camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claus Ibsen (JIRA)" <>
Subject [jira] [Commented] (CAMEL-9735) camel-tarfile throws OutOfMemoryError when splitting large files
Date Tue, 22 Mar 2016 06:03:25 GMT


Claus Ibsen commented on CAMEL-9735:

The CAMEL-NNNN link is to this ticket. What ticket did you mean?

> camel-tarfile throws OutOfMemoryError when splitting large files
> ----------------------------------------------------------------
>                 Key: CAMEL-9735
>                 URL:
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-tarfile
>    Affects Versions: 2.16.2
>            Reporter: Sverker Abrahamsson
> After the change in I started to get
failures with the following stacktrace:
> {noformat}
> Caused by: java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(
>         at
>         at
>         at
>         at
>         at org.apache.camel.util.IOHelper.copy(
>         at org.apache.camel.util.IOHelper.copy(
>         at org.apache.camel.util.IOHelper.copyAndCloseInput(
>         at org.apache.camel.util.IOHelper.copyAndCloseInput(
>         at
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>         at java.lang.reflect.Method.invoke(
>         at org.apache.camel.util.ObjectHelper.invokeMethod(
>         at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(
>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(
>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(
>         at org.apache.camel.dataformat.tarfile.TarIterator.createNextMessage(
>         at org.apache.camel.dataformat.tarfile.TarIterator.tryAdvanceToNext(
>         at org.apache.camel.dataformat.tarfile.TarIterator.hasNext(
>         at org.apache.camel.processor.Splitter$SplitterIterable$1.hasNext(
>         at org.apache.camel.processor.Splitter.updateNewExchange(
> {noformat}
> I.e. using StreamCache makes the whole entry go into memory and since they in my usecase
are several gigabytes it doesn't work very well. I realize that this might be caused since
I don't set any StreamCachingStrategy on my context but reading the corresponding doc part
I get the impression that the default settings should be that it will spool to disk after
reading 128 kbyte.
> However, I really don't want it to do that neither as that would 1) be a significant
overhead to write gigabytes of data to a temp file and 2) I might not even have that much
space where my temp directory is located. Instead the functionality that I want is that the
tarfile entries are streamed and written directly to the destination files without any unnecessary
caching in between.

This message was sent by Atlassian JIRA

View raw message