Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <9ac0c6aa1003050725q77b5708au21dba560a27f8765@mail.gmail.com>
References: <962ea6661003050618m38556f93u5570ec7960528d24@mail.gmail.com>
	 <9ac0c6aa1003050725q77b5708au21dba560a27f8765@mail.gmail.com>
Date: Mon, 8 Mar 2010 13:29:48 +0200
Message-ID: <962ea6661003080329w4ad53612ma72280a1f9802c15@mail.gmail.com>
Subject: Re: IndexWriter.applyDeletes performance
From: Bogdan Ghidireac <bogdan@ecstend.com>
To: java-dev@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Mike,

>
> But... how long does step 2 take? =A0Is it an option to not commit on
> every update? =A0How many docs do you typically update?

I do not commit on every update, I call commit once every 10k
documents. Indexing 10k docs takes around 10 secs.


>
> If you are committing only so that an outside reader can reopen, you
> should consider just using an NRT reader instead (assuming the reader
> is in same JVM as IndexWriter).

My service is just an indexer, I don't need a reader. The new segments
are pushed to a searcher box after each commit.


>
> Roughly how much more RAM consumption do you see when you force pooling?

pooling not forced -> memory after explicit GC: 50 MB
pooling forced -> memory after explicit GC: 250MB

Thank you for opening the JIRA issue.

Bogdan


>
> Mike
>
> On Fri, Mar 5, 2010 at 9:18 AM, Bogdan Ghidireac <bogdan@ecstend.com> wro=
te:
>> Hi,
>>
>> I have an index with 100 million docs that has around 20GB on disk and
>> an update rate of few hundred docs per minute. The new docs are
>> grouped in batches and indexed once every few minutes. My problem is
>> that the update performance degraded too much over time as the index
>> increased in size (distinct docs).
>>
>> My indexing flow looks like this ..
>>
>> 0. create indexWriter (only once)
>> 1. get the open indexWriter
>> 2. for each doc call indexWriter.updateDocument(pkTerm, doc)
>> 3. indexWriter.commit
>> 4. indexWriter.waitForMerges
>> 5. wait for new docs and goto 1.
>>
>> I ran a profiler for several minutes and I noticed that most of the
>> time the indexer is busy applying the deletes. This takes so much time
>> because all terms are loaded for every commit (see the attached
>> profiler screenshot).
>>
>> The index writer has a pool or readers but they are not used unless
>> near real time is enabled. I changed my code to force the pool to be
>> used but the only way I can do this is to request a reader that is
>> never used writer.getReader(). Of course, the memory consumption is
>> higher now because I have terms in memory but the steps 3+4 compete in
>> 1-2 secs compared to 8-10 secs.
>>
>> Is is possible to enable the readers pool at the IndexWriter
>> constructor level? My current method looks like a hack ...
>> I am using Lucene 2.9.2. on Linux.
>>
>> Regards,
>> Bogdan
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org