lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shai Erera (JIRA)" <>
Subject [jira] Commented: (LUCENE-1591) Enable bzip compression in benchmark
Date Mon, 13 Apr 2009 10:21:14 GMT


Shai Erera commented on LUCENE-1591:

bq. Does this issue depend on LUCENE-1595?

No, the other way around. Well ... it's not an actual dependency, just that 1595 will touch
a lot of files, and I want to minimize the noise of working on two issues that touch the same
files (1595 will touch all the files this one touches) simultaneously. It's just a matter
of convenience ...

Besides, I don't see what else can be done as part of this issue. The performance is reasonable,
the code is quite simple. The patch includes some more enhancements to those files that is
unrelated to bzip per sei, but are still required.

BTW, I successfully executed indexLineFile.alg on the 20070527 one-line bz2 file and the overall
indexing process ended in 1h, which seems reasonable to me.

Regarding Apache Compress, I asked the same question, so it's not fair to return it with a
question ;). I don't think we should decide that now. It can be changed in 1595 if we think
Compress is the better approach. Personally I prefer the ant jar, even though I realize it's
adding a large dependency for just 3-4 classes ...

> Enable bzip compression in benchmark
> ------------------------------------
>                 Key: LUCENE-1591
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: contrib/benchmark
>            Reporter: Shai Erera
>             Fix For: 2.9
>         Attachments: ant-1.7.1.jar, LUCENE-1591.patch, LUCENE-1591.patch
> bzip compression can aid the benchmark package by not requiring extracting bzip files
(such as enwiki) in order to index them. The plan is to add a config parameter bzip.compression=true/false
and in the relevant tasks either decompress the input file or compress the output file using
the bzip streams.
> It will add a dependency on ant.jar which contains two classes similar to GZIPOutputStream
and GZIPInputStream which compress/decompress files using the bzip algorithm.
> bzip is known to be superior in its compression performance to the gzip algorithm (~20%
better compression), although it does the compression/decompression a bit slower.
> I wil post a patch which adds this parameter and implement it in LineDocMaker, EnwikiDocMaker
and WriteLineDoc task. Maybe even add the capability to DocMaker or some of the super classes,
so it can be inherited by all sub-classes.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message