commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefan Bodewig (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (COMPRESS-358) Offset is larger than block size in IWA dialect of FramedSnappyCompressorInputStream
Date Tue, 10 Jan 2017 06:18:58 GMT

    [ https://issues.apache.org/jira/browse/COMPRESS-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814044#comment-15814044
] 

Stefan Bodewig commented on COMPRESS-358:
-----------------------------------------

We use an internal lock size of 32k and this one is only configurable for the raw stream,
It seems the IWA files use offsets larger than that. https://github.com/evernote/iwana/blob/master/iwana-extract/src/main/java/com/evernote/iwana/SnappyNoCRCFramedInputStream.java#L33
uses 128k which is way bigger than the size the "spec" talks about

{quote}
Note that the data portion of the chunk, i.e., the compressed contents,
can be at most 16777211 bytes (2^24 - 1, minus the checksum).
However, we place an additional restriction that the uncompressed data
in a chunk must be no longer than 65536 bytes. This allows consumers to
easily use small fixed-size buffers.
{quote}

Yet another deviation, I guess.

I'll make the block size configurable for the framed streams as well, probably won't manage
to do so today, though.

> Offset is larger than block size in IWA dialect of FramedSnappyCompressorInputStream
> ------------------------------------------------------------------------------------
>
>                 Key: COMPRESS-358
>                 URL: https://issues.apache.org/jira/browse/COMPRESS-358
>             Project: Commons Compress
>          Issue Type: Bug
>          Components: Compressors
>    Affects Versions: 1.12
>            Reporter: Tim Allison
>            Priority: Trivial
>         Attachments: DocumentStyleSheet_commons_unsnapped.iwa, DocumentStyleSheet_fromZipStream.iwa,
DocumentStylesheet.iwa, DocumentStylesheet_iwana_unsnapped.iwa, DocumentStylesheet_uncompressed.iwa,
testNumbersNew.numbers
>
>
> I finally was able to run the FramedSnappyCompressorInputStreamon a larger number of
.iwa files.  I got the following on two files:
> {noformat}
> java.io.IOException: Offset is larger than block size
> 	at org.apache.commons.compress.compressors.snappy.SnappyCompressorInputStream.expandCopy(SnappyCompressorInputStream.java:341)
> 	at org.apache.commons.compress.compressors.snappy.SnappyCompressorInputStream.fill(SnappyCompressorInputStream.java:212)
> 	at org.apache.commons.compress.compressors.snappy.SnappyCompressorInputStream.read(SnappyCompressorInputStream.java:134)
> 	at org.apache.commons.compress.compressors.snappy.FramedSnappyCompressorInputStream.readOnce(FramedSnappyCompressorInputStream.java:166)
> 	at org.apache.commons.compress.compressors.snappy.FramedSnappyCompressorInputStream.read(FramedSnappyCompressorInputStream.java:122)
> 	at java.io.InputStream.read(InputStream.java:101)
> 	at java.nio.file.Files.copy(Files.java:2908)
> 	at java.nio.file.Files.copy(Files.java:3027)
> {noformat}
> No good deed goes unpunished... :)  Thank you, again!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message