camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claus Ibsen (JIRA)" <>
Subject [jira] [Resolved] (CAMEL-9735) camel-tarfile throws OutOfMemoryError when splitting large files
Date Tue, 22 Mar 2016 16:02:25 GMT


Claus Ibsen resolved CAMEL-9735.
    Resolution: Fixed

Thanks for the details. We are closing this ticket as the OOME is fixed again.

We need another ticket for the tar splitter may spit out 1 extra which becomes a null message.

> camel-tarfile throws OutOfMemoryError when splitting large files
> ----------------------------------------------------------------
>                 Key: CAMEL-9735
>                 URL:
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-tarfile
>    Affects Versions: 2.16.2, 2.17.0
>            Reporter: Sverker Abrahamsson
>            Assignee: Claus Ibsen
>            Priority: Minor
>             Fix For: 2.16.3, 2.18.0, 2.17.1
> After the change in I started to get
failures with the following stacktrace:
> {noformat}
> Caused by: java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(
>         at
>         at
>         at
>         at
>         at org.apache.camel.util.IOHelper.copy(
>         at org.apache.camel.util.IOHelper.copy(
>         at org.apache.camel.util.IOHelper.copyAndCloseInput(
>         at org.apache.camel.util.IOHelper.copyAndCloseInput(
>         at
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>         at java.lang.reflect.Method.invoke(
>         at org.apache.camel.util.ObjectHelper.invokeMethod(
>         at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(
>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(
>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(
>         at org.apache.camel.dataformat.tarfile.TarIterator.createNextMessage(
>         at org.apache.camel.dataformat.tarfile.TarIterator.tryAdvanceToNext(
>         at org.apache.camel.dataformat.tarfile.TarIterator.hasNext(
>         at org.apache.camel.processor.Splitter$SplitterIterable$1.hasNext(
>         at org.apache.camel.processor.Splitter.updateNewExchange(
> {noformat}
> I.e. using StreamCache makes the whole entry go into memory and since they in my usecase
are several gigabytes it doesn't work very well. I realize that this might be caused since
I don't set any StreamCachingStrategy on my context but reading the corresponding doc part
I get the impression that the default settings should be that it will spool to disk after
reading 128 kbyte.
> However, I really don't want it to do that neither as that would 1) be a significant
overhead to write gigabytes of data to a temp file and 2) I might not even have that much
space where my temp directory is located. Instead the functionality that I want is that the
tarfile entries are streamed and written directly to the destination files without any unnecessary
caching in between.

This message was sent by Atlassian JIRA

View raw message