Mailing-List: contact issues-help@commons.apache.org; run by ezmlm
Precedence: bulk
Reply-To: issues@commons.apache.org
Date: Tue, 10 Jan 2017 06:18:58 +0000 (UTC)
From: "Stefan Bodewig (JIRA)" <jira@apache.org>
To: issues@commons.apache.org
Message-ID: <JIRA.12982074.1466624710000.693298.1484029138369@Atlassian.JIRA>
In-Reply-To: <JIRA.12982074.1466624710000@Atlassian.JIRA>
References: <JIRA.12982074.1466624710000@Atlassian.JIRA> <JIRA.12982074.1466624710689@arcas>
Subject: [jira] [Commented] (COMPRESS-358) Offset is larger than block size
 in IWA dialect of FramedSnappyCompressorInputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 10 Jan 2017 06:19:00 -0000


    [ https://issues.apache.org/jira/browse/COMPRESS-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814044#comment-15814044 ] 

Stefan Bodewig commented on COMPRESS-358:
-----------------------------------------

We use an internal lock size of 32k and this one is only configurable for the raw stream, It seems the IWA files use offsets larger than that. https://github.com/evernote/iwana/blob/master/iwana-extract/src/main/java/com/evernote/iwana/SnappyNoCRCFramedInputStream.java#L33 uses 128k which is way bigger than the size the "spec" talks about

{quote}
Note that the data portion of the chunk, i.e., the compressed contents,
can be at most 16777211 bytes (2^24 - 1, minus the checksum).
However, we place an additional restriction that the uncompressed data
in a chunk must be no longer than 65536 bytes. This allows consumers to
easily use small fixed-size buffers.
{quote}

Yet another deviation, I guess.

I'll make the block size configurable for the framed streams as well, probably won't manage to do so today, though.

> Offset is larger than block size in IWA dialect of FramedSnappyCompressorInputStream
> ------------------------------------------------------------------------------------
>
>                 Key: COMPRESS-358
>                 URL: https://issues.apache.org/jira/browse/COMPRESS-358
>             Project: Commons Compress
>          Issue Type: Bug
>          Components: Compressors
>    Affects Versions: 1.12
>            Reporter: Tim Allison
>            Priority: Trivial
>         Attachments: DocumentStyleSheet_commons_unsnapped.iwa, DocumentStyleSheet_fromZipStream.iwa, DocumentStylesheet.iwa, DocumentStylesheet_iwana_unsnapped.iwa, DocumentStylesheet_uncompressed.iwa, testNumbersNew.numbers
>
>
> I finally was able to run the FramedSnappyCompressorInputStreamon a larger number of .iwa files.  I got the following on two files:
> {noformat}
> java.io.IOException: Offset is larger than block size
> 	at org.apache.commons.compress.compressors.snappy.SnappyCompressorInputStream.expandCopy(SnappyCompressorInputStream.java:341)
> 	at org.apache.commons.compress.compressors.snappy.SnappyCompressorInputStream.fill(SnappyCompressorInputStream.java:212)
> 	at org.apache.commons.compress.compressors.snappy.SnappyCompressorInputStream.read(SnappyCompressorInputStream.java:134)
> 	at org.apache.commons.compress.compressors.snappy.FramedSnappyCompressorInputStream.readOnce(FramedSnappyCompressorInputStream.java:166)
> 	at org.apache.commons.compress.compressors.snappy.FramedSnappyCompressorInputStream.read(FramedSnappyCompressorInputStream.java:122)
> 	at java.io.InputStream.read(InputStream.java:101)
> 	at java.nio.file.Files.copy(Files.java:2908)
> 	at java.nio.file.Files.copy(Files.java:3027)
> {noformat}
> No good deed goes unpunished... :)  Thank you, again!


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)