lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: recurrent IO/CPU peaks
Date Wed, 19 Jan 2011 13:37:02 GMT
This is normal behavior, unfortunately.

The default LogByteSizeMergePolicy does mergeFactor (default 10) small
merges in a row, then must do a 10X bigger merge.  Every 100 small
merges it does a 100X bigger merge, etc.

You can try using the BalancedSegmentMergePolicy (in contrib/misc) --
it tries to reduce the large merges, and never "inadvertently
optimizes" like Lucene's default merge policy.

If you do use it, please report back... we are tempted to improve the
default merge policy by either slurping BSMP into core, or, borrowing
it's approach...

If you are on Linux, you could also try using the
DirectIOLinuxDirectory, which avoid messing up the OS's buffer cache
due to merging.  But that dir impl is experimental, and you can't just
"use" it across the board (you should use it only for IndexWriter, and
possibly only when merges are kicking off).

Mike

On Wed, Jan 19, 2011 at 2:31 AM,  <v.sevel@lombardodier.com> wrote:
>
> Hi,
>
> I have an application that continously indexes 140 documents/s (we commit
> after each second) using lucene 2.9. at the beginning of the test the index
> is empty. during the test, I monitored this application, specifically in
> terms of IOs and CPU.
>
> what I have seen, is that, on a regular basis (a few minutes), there are
> small peaks both for IOs and CPU, and once in a while, there is a much
> higher peak. the small peaks are small enough that do not disrupt the
> system, however, the high peak tends to steal some resources out of the
> other processes, and gets disruptive when the machine is loaded.
>
> I logged the internal activity of the lucene writer during the high peak,
> and what I have seen that could correlate is the number of merge documents
> that suddenly grows, and executes shortly after another merge:
>
>         Line 131: 17:46:26.268 IW 0 [Lucene Merge Thread #215]: merge: total
> 1391 docs
>         Line 539: 17:46:36.449 IW 0 [Lucene Merge Thread #216]: merge: total
> 1462 docs
>         Line 570: 17:46:36.565 IW 0 [Lucene Merge Thread #216]: merge: total
> 13753 docs
>         Line 597: 17:46:37.381 IW 0 [Lucene Merge Thread #216]: merge: total
> 171593 docs
>         Line 991: 17:46:46.503 IW 0 [Lucene Merge Thread #217]: merge: total
> 1537 docs
>         Line 1439: 17:46:57.602 IW 0 [Lucene Merge Thread #218]: merge:
> total 1710 docs
>         Line 1874: 17:47:07.680 IW 0 [Lucene Merge Thread #219]: merge:
> total 1532 docs
>         Line 2282: 17:47:17.804 IW 0 [Lucene Merge Thread #220]: merge:
> total 1555 docs
>
>
> Assuming this is a normal behavior, are there ways to minimize the amplitude
> of the high peaks?
>
> I am attaching the lucene writer log and the systar report.
>
> Thanks for the help.
>
>
>
>
> Vincent
>
>
>
>
> ************************ DISCLAIMER ************************
> This message is intended only for use by the person to
> whom it is addressed. It may contain information that is
> privileged and confidential. Its content does not
> constitute a formal commitment by Lombard Odier
> Darier Hentsch & Cie or any of its branches or affiliates.
> If you are not the intended recipient of this message,
> kindly notify the sender immediately and destroy this
> message. Thank You.
> *****************************************************************
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message