Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <00cd01cbc88b$4771f0d0$d655d270$@com>
References: <00cd01cbc88b$4771f0d0$d655d270$@com>
Date: Wed, 9 Feb 2011 18:14:45 -0500
Message-ID: <AANLkTinQ8en2J3TQPjuxE2CJcqizCoqN7wnWy1UJ9qG1@mail.gmail.com>
Subject: Re: index size doubling / optimization (Lucene 3.0.3)
From: Michael McCandless <lucene@mikemccandless.com>
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1

This is not expected.

Did the last IW exit "gracefully"?  If so, it should delete the old
segments after swapping in the optimized one.

Can you post infoStream output after running optimize?

Mike

On Wed, Feb 9, 2011 at 1:58 PM, Phil Herold <pherold@d-wise.com> wrote:
> I know that the size of a Lucene index can double while optimization is
> underway, but it's supposed to eventually settle back down to the original
> size, correct? We have a Lucene index consisting of 100K documents, that is
> normally about 12GB in size. It is split across 10 sub-indexes which we
> search using MultiSearcher. It takes our system about 7 hours to traverse
> the file system and update the index, which typically adds, updates or
> deletes anywhere from a dozen to a few hundred documents. We optimize each
> sub-index at the end (although this is configurable). The system seems to
> run fine for several days, with the total size of the index staying fairly
> consistent, then all of the sudden the index size doubles to about 25GB, and
> stays there. I'm assuming this is happening after an optimization-there is
> certainly not a doubling of the data that is being added.
>
>
>
> Is this expected or known behavior, or a bug of some kind?
>
>
>
> I've read various postings on the 'net regarding optimization, and when to
> do it, if at all, and I'm certainly open to other strategies. Search time is
> critical for our users.
>
>
>
> FWIW, we have the following tunable parameters configured for our index:
>
>
>
> mergeFactor: 5
>
> maxMergeDocs: 1000
>
> maxBufferedDocs: 200
>
> RAMBufferSizeMB: 16
>
>
>
> Any advice or help is appreciated.
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org