Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\))
Subject: Re: DocValues formats hold large byte[][]s even when using
 MMapDirectory
From: Steven Schlansker <steven@likeness.com>
In-Reply-To: 
 <CAL8PwkaNsOiPkkOeZ+ajL+P4tBmJdTCEqK9CT9n5LpvyMzSuJA@mail.gmail.com>
Date: Wed, 2 Oct 2013 11:37:57 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <BB8DADD5-97E6-47AA-AA16-8E55CC635852@likeness.com>
References: <FCD29CA0-B66B-4DD4-A5DA-DC45AD011E02@likeness.com>
 <CAL8PwkaNsOiPkkOeZ+ajL+P4tBmJdTCEqK9CT9n5LpvyMzSuJA@mail.gmail.com>
To: java-user@lucene.apache.org


On Oct 2, 2013, at 11:16 AM, Michael McCandless =
<lucene@mikemccandless.com> wrote:

> In Lucene 4.5 (coming out any day now) we've switched by default to a
> "mostly on disk" impl for doc values.
>=20

Awesome!  Looking forward to that then.

> Before that, you can use DiskDocValuesFormat instead.
>=20
> But you'll need to re-index (or create a new index and use
> IW.addIndexes) to cutover your current index to the DiskDVFormat.
>=20

I see a few references scattered on the internet but it's not in my =
Lucene jars.  The one reference I saw to it indicated that every patch =
release of Lucene will require a full reindex when using this, which is =
a serious bummer.

So I think I'll hold out for 4.5 and hope that that solves my problem.
Thanks for the help!


>=20
> On Wed, Oct 2, 2013 at 2:11 PM, Steven Schlansker =
<steven@likeness.com> wrote:
>> Hi,
>>=20
>> I have a search application using Lucene 4.4.0 with various =
BinaryDocValues and SortedSetDocValues.
>> We use MMapDirectory to help keep the Java heap small / GC pause =
times short and instead rely on the OS buffer cache to keep things fast, =
which I gather is generally considered a "best practice" around here.
>> As our index grows, I've noticed that we are getting GC pauses and =
later OOM errors when reloading a new index due to gigabytes of =
byte[][]s held by Lucene42DocValuesProducer, specifically the =
PagedBytes.Reader.blocks from within =
Lucene42DocValuesProducer.loadBinary
>>=20
>> I would have expected DocValues fields to use mapped bytes instead of =
copying into the Java heap much as the "main" index data is.  Is this a =
technical limitation, a "we haven't gotten there yet" feature request, =
or something different entirely?
>>=20
>> Thanks for helping my understanding,
>> Steven
>>=20
>>=20
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>=20
>=20
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>=20


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org