lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: Analyzing performance and memory consumption for boolean queries
Date Wed, 24 Jun 2009 07:33:08 GMT
> 1. For search time to vary from < 1 second => 20 seconds, the only
> two things I've seen are:
> 
> * Serious JVM garbage collection problems.
> * You're in Linux swap hell.
> 
> We tracked similar issued down by creating a testbed that let us run
> a set of real-world queries, such that we could trigger these types
> of problems when we had appropriate instrumentation on and recording.

I had similar problems with our configuration, too. Suddenly sometimes the
server even did not respond. The problem was (I think is the same here): the
GC. The standard Java GC is not multithreaded, so if you have lots of
traffic at some time, the JVM halts all threads and starts to GC, which can
take very long time with so big heap sizes.

On our server with indexes of similar disk space size (not documents), I
changed the JVM options to use:

-Xms4096M -Xmx8192M -XX:MaxPermSize=512M -Xrs -XX:+UseConcMarkSweepGC
-XX:+UseParNewGC -verbosegc -XX:+PrintGCDetails -XX:+UseLargePages

This also turns on GC debugging and the ParNewGC and ConcMarkSweepGC works
much better here (but please do not simply copy these settings, read about
them in the JVM docs, exact settings depend on your use-case!). I had no
hangs anymore since this change. The JVM prints information about garbage
collection to stderr (which you should study, there is a paper from sun
about it). Our web server (Sun Java System Webserver 7.0, Solaris 10 x64)
also reports the time used in complete for GC, during a server uptime of 11
days it used about 4 hours to GC in parallel threads. This config works good
with multiple CPUs, in our case, one could say: "one CPU is GCing the whole
time" :-)

There is also a howto on the lucid imagination page about different GCs and
Lucene.

Uwe


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message