camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sverker Abrahamsson (JIRA)" <>
Subject [jira] [Commented] (CAMEL-9735) camel-tarfile throws OutOfMemoryError when splitting large files
Date Tue, 22 Mar 2016 12:23:25 GMT


Sverker Abrahamsson commented on CAMEL-9735:

The same race condition should happen also with ZipIterator, as I copied the logic from there
when I made the tarfile component.

> camel-tarfile throws OutOfMemoryError when splitting large files
> ----------------------------------------------------------------
>                 Key: CAMEL-9735
>                 URL:
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-tarfile
>    Affects Versions: 2.16.2, 2.17.0
>            Reporter: Sverker Abrahamsson
>            Assignee: Claus Ibsen
>            Priority: Minor
>             Fix For: 2.16.3, 2.18.0, 2.17.1
> After the change in I started to get
failures with the following stacktrace:
> {noformat}
> Caused by: java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(
>         at
>         at
>         at
>         at
>         at org.apache.camel.util.IOHelper.copy(
>         at org.apache.camel.util.IOHelper.copy(
>         at org.apache.camel.util.IOHelper.copyAndCloseInput(
>         at org.apache.camel.util.IOHelper.copyAndCloseInput(
>         at
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>         at java.lang.reflect.Method.invoke(
>         at org.apache.camel.util.ObjectHelper.invokeMethod(
>         at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(
>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(
>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(
>         at org.apache.camel.dataformat.tarfile.TarIterator.createNextMessage(
>         at org.apache.camel.dataformat.tarfile.TarIterator.tryAdvanceToNext(
>         at org.apache.camel.dataformat.tarfile.TarIterator.hasNext(
>         at org.apache.camel.processor.Splitter$SplitterIterable$1.hasNext(
>         at org.apache.camel.processor.Splitter.updateNewExchange(
> {noformat}
> I.e. using StreamCache makes the whole entry go into memory and since they in my usecase
are several gigabytes it doesn't work very well. I realize that this might be caused since
I don't set any StreamCachingStrategy on my context but reading the corresponding doc part
I get the impression that the default settings should be that it will spool to disk after
reading 128 kbyte.
> However, I really don't want it to do that neither as that would 1) be a significant
overhead to write gigabytes of data to a temp file and 2) I might not even have that much
space where my temp directory is located. Instead the functionality that I want is that the
tarfile entries are streamed and written directly to the destination files without any unnecessary
caching in between.

This message was sent by Atlassian JIRA

View raw message