Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Fri, 26 Oct 2012 20:01:13 +0000 (UTC)
From: "Yonik Seeley (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: <2025343724.33279.1351281673812.JavaMail.jiratomcat@arcas>
In-Reply-To: <1760127848.32806.1351274712273.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (LUCENE-4509) Make CompressingStoredFieldsFormat
 the new default StoredFieldsFormat impl
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/LUCENE-4509?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D134=
85154#comment-13485154 ]=20

Yonik Seeley commented on LUCENE-4509:
--------------------------------------

Nice timing Adrien... I was just going to ask how we could enable this easi=
est in Solr (or if it should in fact be the default).

One data point: 100GB of compressed stored fields =3D=3D 6.25M index entrie=
s =3D=3D 75MB RAM
That seems decent for a default.

               =20
> Make CompressingStoredFieldsFormat the new default StoredFieldsFormat imp=
l
> -------------------------------------------------------------------------=
-
>
>                 Key: LUCENE-4509
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4509
>             Project: Lucene - Core
>          Issue Type: Wish
>          Components: core/store
>            Reporter: Adrien Grand
>            Priority: Minor
>
> What would you think of making CompressingStoredFieldsFormat the new defa=
ult StoredFieldsFormat?
> Stored fields compression has many benefits=C2=A0:
>  - it makes the I/O cache work for us,
>  - file-based index replication/backup becomes cheaper.
> Things to know:
>  - even with incompressible data, there is less than 0.5% overhead with L=
Z4,
>  - LZ4 compression requires ~ 16kB of memory and LZ4 HC compression requi=
res ~ 256kB,
>  - LZ4 uncompression has almost no memory overhead,
>  - on my low-end laptop, the LZ4 impl in Lucene uncompresses at ~ 300mB/s=
.
> I think we could use the same default parameters as in CompressingCodec :
>  - LZ4 compression,
>  - in-memory stored fields index that is very memory-efficient (less than=
 12 bytes per block of compressed docs) and uses binary search to locate do=
cuments in the fields data file,
>  - 16 kB blocks (small enough so that there is no major slow down when th=
e whole index would fit into the I/O cache anyway, and large enough to prov=
ide interesting compression ratios=C2=A0; for example Robert got a 0.35 com=
pression ratio with the geonames.org database).
> Any concerns?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org