commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Bodewig <bode...@apache.org>
Subject Re: [COMPRESS] Anyone implemented "pigz"?
Date Wed, 10 May 2017 07:57:41 GMT
On 2017-05-10, Gary Gregory wrote:

> I think the question is can/should [Compress] use any of the stock code
> in java.util.zip in a multi-threaded fashion for performance gains.

We rely on java.util.zip.Deflater for DEFLATE which isn't thread safe by
itself.

But we could implement the same strategy pigz uses, which is to break up
the stream into chunks and work on the chunks in parallel. Combining the
output of several streams may become tricky using the Java API.

If my first read of the comments in
https://github.com/madler/pigz/blob/master/pigz.c is correct then we'd
need to manipulate the output of Deflater in order to strip headers and
trailers and insert empty stored blocks as well as create headers and
trailers of our own for the combined output.

In theory we could implement something like pigz on top of the LZ77
support I've added for Snappy and LZ4 (and some additional Hufmann code
yet to be written) but it would be slower than zlib - probably a lot -
and likely eat up the speed gain provided by parallel processing.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Mime
View raw message