Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
MIME-Version: 1.0
References: <a43e3fdf-2a4c-a8d4-1fbd-1d08fc4fc209@codetrails.com>
 <CAN4YXvceJgmT6RrbAg4nZFTOk9BWb08Nxr5Z3u=Et89_3yYkdg@mail.gmail.com>
 <CAPsWd+Mex9PXdVHU3HopeRcp30estN1jfVO2mTdRO46csDVj0Q@mail.gmail.com>
 <71b86d19-2d2d-e809-309f-3fb7213ac907@codetrails.com>
In-Reply-To: <71b86d19-2d2d-e809-309f-3fb7213ac907@codetrails.com>
From: Adrien Grand <jpountz@gmail.com>
Date: Tue, 15 Mar 2016 16:41:20 +0000
Message-ID: 
 <CAPsWd+PujBcLTDrMLQKvPHrGJyQiQpZ3egU64kBoq6OsOT=T6A@mail.gmail.com>
Subject: Re: Canonicalize stored fields (small set of possible values)
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=001a11402208bf12f6052e191157

--001a11402208bf12f6052e191157
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Le mar. 15 mars 2016 =C3=A0 17:33, Andreas Sewe <andreas.sewe@codetrails.co=
m> a
=C3=A9crit :

> I am afraid I don't understand. Do you suggest using IntFields as ID
> instead of StringFields, as they are presumably stored more efficiently?
>

Exactly. Integers are stored using zig-zag encoding and variable byte. So
numbers between -64 and 63 use 1 byte, numbers between -8192 and 8191 use 2
bytes, etc.


> > Otherwise, even without doing anything, things
> > should not be too bad thanks to stored fields compression.
>
> AFAICT, the fields are not compressed on disk right now. At least, "grep
> -c" finds my field over and over in the index files.
>
> So, how do I enabled stored fields compression. Googling turned up
> Store.COMPRESS, but that doesn't exist in 5.2.1.
>

Compression is on by default, but we split the stored fields file into
blocks of 16KB and compress each block individually. So each 16KB block
still needs to store values at least once before the compression algorithm
can make references to it.

If you want to enable stronger compression, you can do
`indexWriterConfig.setCodec(new Lucene54Codec(Mode.BEST_COMPRESSION))`
which will use DEFLATE insead of LZ4 to compress blocks. In addition of
removing duplicates like LZ4, DEFLATE also applies some Huffman coding so
that you should see better compression if your field values use some
symbols much more frequently than others.

--001a11402208bf12f6052e191157--