lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject recurrent IO/CPU peaks
Date Wed, 19 Jan 2011 07:31:17 GMT

I have an application that continously indexes 140 documents/s (we commit 
after each second) using lucene 2.9. at the beginning of the test the 
index is empty. during the test, I monitored this application, 
specifically in terms of IOs and CPU.

what I have seen, is that, on a regular basis (a few minutes), there are 
small peaks both for IOs and CPU, and once in a while, there is a much 
higher peak. the small peaks are small enough that do not disrupt the 
system, however, the high peak tends to steal some resources out of the 
other processes, and gets disruptive when the machine is loaded.

I logged the internal activity of the lucene writer during the high peak, 
and what I have seen that could correlate is the number of merge documents 
that suddenly grows, and executes shortly after another merge:

        Line 131: 17:46:26.268 IW 0 [Lucene Merge Thread #215]: merge: 
total 1391 docs
        Line 539: 17:46:36.449 IW 0 [Lucene Merge Thread #216]: merge: 
total 1462 docs
        Line 570: 17:46:36.565 IW 0 [Lucene Merge Thread #216]: merge: 
total 13753 docs
        Line 597: 17:46:37.381 IW 0 [Lucene Merge Thread #216]: merge: 
total 171593 docs
        Line 991: 17:46:46.503 IW 0 [Lucene Merge Thread #217]: merge: 
total 1537 docs
        Line 1439: 17:46:57.602 IW 0 [Lucene Merge Thread #218]: merge: 
total 1710 docs
        Line 1874: 17:47:07.680 IW 0 [Lucene Merge Thread #219]: merge: 
total 1532 docs
        Line 2282: 17:47:17.804 IW 0 [Lucene Merge Thread #220]: merge: 
total 1555 docs

Assuming this is a normal behavior, are there ways to minimize the 
amplitude of the high peaks?

I am attaching the lucene writer log and the systar report.

Thanks for the help.


************************ DISCLAIMER ************************
This message is intended only for use by the person to
whom it is addressed. It may contain information that is
privileged and confidential. Its content does not
constitute a formal commitment by Lombard Odier
Darier Hentsch & Cie or any of its branches or affiliates.
If you are not the intended recipient of this message,
kindly notify the sender immediately and destroy this
message. Thank You.

View raw message