commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Sicker <boa...@gmail.com>
Subject Re: [COMPRESS] Anyone implemented "pigz"?
Date Wed, 10 May 2017 15:09:45 GMT
Would the scattering and gathering byte channel APIs in java.nio be helpful
in splitting up a stream into chunks for parallel processing?

On 10 May 2017 at 02:57, Stefan Bodewig <bodewig@apache.org> wrote:

> On 2017-05-10, Gary Gregory wrote:
>
> > I think the question is can/should [Compress] use any of the stock code
> > in java.util.zip in a multi-threaded fashion for performance gains.
>
> We rely on java.util.zip.Deflater for DEFLATE which isn't thread safe by
> itself.
>
> But we could implement the same strategy pigz uses, which is to break up
> the stream into chunks and work on the chunks in parallel. Combining the
> output of several streams may become tricky using the Java API.
>
> If my first read of the comments in
> https://github.com/madler/pigz/blob/master/pigz.c is correct then we'd
> need to manipulate the output of Deflater in order to strip headers and
> trailers and insert empty stored blocks as well as create headers and
> trailers of our own for the combined output.
>
> In theory we could implement something like pigz on top of the LZ77
> support I've added for Snappy and LZ4 (and some additional Hufmann code
> yet to be written) but it would be slower than zlib - probably a lot -
> and likely eat up the speed gain provided by parallel processing.
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>


-- 
Matt Sicker <boards@gmail.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message