Mailing-List: contact issues-help@commons.apache.org; run by ezmlm
Precedence: bulk
Reply-To: issues@commons.apache.org
Date: Fri, 14 Apr 2017 16:03:42 +0000 (UTC)
From: "Stefan Bodewig (JIRA)" <jira@apache.org>
To: issues@commons.apache.org
Message-ID: <JIRA.13040268.1486161491000.285183.1492185822348@Atlassian.JIRA>
In-Reply-To: <JIRA.13040268.1486161491000@Atlassian.JIRA>
References: <JIRA.13040268.1486161491000@Atlassian.JIRA> <JIRA.13040268.1486161491332@jira-lw-us.apache.org>
Subject: [jira] [Commented] (COMPRESS-382) OutOfMemoryError from
 CompressorStreamFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 14 Apr 2017 16:03:48 -0000


    [ https://issues.apache.org/jira/browse/COMPRESS-382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969201#comment-15969201 ] 

Stefan Bodewig commented on COMPRESS-382:
-----------------------------------------

Some of the streams support parametrizations, like the window size for Snappy, the compression level of gzip and so on. So far we pick something as a default and this is what you get when using the factory. The current thinking behind this is that if you know you want certain parameters, then you know which format you want and don't need to go through the factory anyway.

I'm not sure whether we can actually enforce thresholds for all of the formats. If we can, then it may make sense to add it as a parameter to the factory. Unfortunately the interface approach of ServiceLoader makes it way more difficult to evolve the factory's contract.


> OutOfMemoryError from CompressorStreamFactory
> ---------------------------------------------
>
>                 Key: COMPRESS-382
>                 URL: https://issues.apache.org/jira/browse/COMPRESS-382
>             Project: Commons Compress
>          Issue Type: Bug
>          Components: Compressors
>    Affects Versions: 1.10, 1.11, 1.12
>         Environment: Windows7, jre1.8.0_101 x64
>            Reporter: Luis Filipe Nassif
>         Attachments: data.mui
>
>
> While using Tika-1.14 to detect file types, the attached 1KB file triggered an OOME with 1GB heap. Tika calls CompressorStreamFactory.createCompressorInputStream(in) to detect if the file is a compressor stream, but CompressorStreamFactory erroneously detects it as a LZMACompressorInputStream and when the LZMACompressorInputStream is instanciated the OOME is thrown. This error does not happen with commons-compress versions prior to 1.10, when auto detecting LZMA streams was added. OOME stacktrace below:
> {code}
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 	at org.tukaani.xz.lz.LZDecoder.<init>(Unknown Source) ~[xz-1.5.jar:1.5]
> 	at org.tukaani.xz.LZMAInputStream.initialize(Unknown Source) ~[xz-1.5.jar:1.5]
> 	at org.tukaani.xz.LZMAInputStream.initialize(Unknown Source) ~[xz-1.5.jar:1.5]
> 	at org.tukaani.xz.LZMAInputStream.<init>(Unknown Source) ~[xz-1.5.jar:1.5]
> 	at org.tukaani.xz.LZMAInputStream.<init>(Unknown Source) ~[xz-1.5.jar:1.5]
> 	at org.apache.commons.compress.compressors.lzma.LZMACompressorInputStream.<init>(LZMACompressorInputStream.java:48) ~[commons-compress-1.10.jar:1.10]
> 	at org.apache.commons.compress.compressors.CompressorStreamFactory.createCompressorInputStream(CompressorStreamFactory.java:251) ~[commons-compress-1.10.jar:1.10]
> 	at org.apache.tika.parser.pkg.ZipContainerDetector.detectCompressorFormat(ZipContainerDetector.java:109) ~[tika-parsers-1.14.jar:1.14]
> 	at org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:95) ~[tika-parsers-1.14.jar:1.14]
> 	at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77) ~[tika-core-1.14.jar:1.14]
> 	at dpf.sp.gpinf.indexer.process.task.SignatureTask.process(SignatureTask.java:50) ~[iped.jar:?]
> 	at dpf.sp.gpinf.indexer.process.task.AbstractTask.processMonitorTimeout(AbstractTask.java:203) ~[iped.jar:?]
> 	at dpf.sp.gpinf.indexer.process.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:152) ~[iped.jar:?]
> 	at dpf.sp.gpinf.indexer.process.task.AbstractTask.sendToNextTask(AbstractTask.java:190) ~[iped.jar:?]
> 	at dpf.sp.gpinf.indexer.process.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:160) ~[iped.jar:?]
> 	at dpf.sp.gpinf.indexer.process.task.AbstractTask.sendToNextTask(AbstractTask.java:190) ~[iped.jar:?]
> 	at dpf.sp.gpinf.indexer.process.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:160) ~[iped.jar:?]
> 	at dpf.sp.gpinf.indexer.process.task.AbstractTask.sendToNextTask(AbstractTask.java:190) ~[iped.jar:?]
> 	at dpf.sp.gpinf.indexer.process.task.AbstractTask.processAndSendToNextTask(AbstractTask.java:160) ~[iped.jar:?]
> 	at dpf.sp.gpinf.indexer.process.Worker.process(Worker.java:174) ~[iped.jar:?]
> 	... 1 more
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)