commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Sicker <>
Subject Re: [COMPRESS] Anyone implemented "pigz"?
Date Wed, 10 May 2017 15:09:45 GMT
Would the scattering and gathering byte channel APIs in java.nio be helpful
in splitting up a stream into chunks for parallel processing?

On 10 May 2017 at 02:57, Stefan Bodewig <> wrote:

> On 2017-05-10, Gary Gregory wrote:
> > I think the question is can/should [Compress] use any of the stock code
> > in in a multi-threaded fashion for performance gains.
> We rely on for DEFLATE which isn't thread safe by
> itself.
> But we could implement the same strategy pigz uses, which is to break up
> the stream into chunks and work on the chunks in parallel. Combining the
> output of several streams may become tricky using the Java API.
> If my first read of the comments in
> is correct then we'd
> need to manipulate the output of Deflater in order to strip headers and
> trailers and insert empty stored blocks as well as create headers and
> trailers of our own for the combined output.
> In theory we could implement something like pigz on top of the LZ77
> support I've added for Snappy and LZ4 (and some additional Hufmann code
> yet to be written) but it would be slower than zlib - probably a lot -
> and likely eat up the speed gain provided by parallel processing.
> Stefan
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

Matt Sicker <>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message