commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lasse Collin <>
Subject [compress] XZ support and inconsistencies in the existing compressors
Date Wed, 03 Aug 2011 19:22:44 GMT

I have been working on XZ data compression implementation in Java
<>. I was told that it could be nice
to get XZ support into Commons Compress.

I looked at the APIs and code in Commons Compress to see how XZ
support could be added. I was especially looking for details where
one would need to be careful to make different compressors behave
consistently compared to each other. I found a few possible problems
in the existing code:

(1) CompressorOutputStream should have finish(). Now
    BZip2CompressorOutputStream has finish() but
    GzipCompressorOutputStream doesn't. This should be easy to
    fix because supports finish().

(2) BZip2CompressorOutputStream.flush() calls out.flush() but it
    doesn't flush data buffered by BZip2CompressorOutputStream.
    Thus not all data written to the Bzip2 stream will be available
    in the underlying output stream after flushing. This kind of
    flush() implementation doesn't seem very useful.

    GzipCompressorOutputStream.flush() is the default version
    from InputStream and thus does nothing. Adding flush()
    into GzipCompressorOutputStream is hard because and don't
    support sync flushing before Java 7. To get Gzip flushing in
    older Java versions one might need a complete reimplementation
    of the Deflate algorithm which isn't necessarily practical.

(3) BZip2CompressorOutputStream has finalize() that finishes a stream
    that hasn't been explicitly finished or closed. This doesn't seem
    useful. GzipCompressorOutputStream doesn't have an equivalent

(4) The decompressor streams don't support concatenated .gz and .bz2
    files. This can be OK when compressed data is used inside another
    file format or protocol, but with regular (standalone) .gz and
    .bz2 files it is bad to stop after the first compressed stream
    and silently ignore the remaining compressed data.

    Fixing this in BZip2CompressorInputStream should be relatively
    easy because it stops right after the last byte of the compressed
    stream. Fixing GzipCompressorInputStream is harder because the
    problem is inherited from
    which reads input past the end of the first stream. One
    might need to reimplement .gz container support on top of or

The XZ compressor supports finish() and flush(). The XZ decompressor
supports concatenated .xz files, but there is also a single-stream
version that behaves similarly to the current version of

Assuming that there will be some interest in adding XZ support into
Commons Compress, is it OK make Commons Compress depend on the XZ
package org.tukaani.xz, or should the XZ code be modified so that
it could be included as an internal part in Commons Compress? I
would prefer depending on org.tukaani.xz because then there is
just one code base to keep up to date.

Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message