lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Brackett" <nbrack...@net-temps.com>
Subject Search deadlocking under load
Date Fri, 08 Jul 2005 15:05:55 GMT
Hey all,

We're looking to use Lucene as the back end to our website and we're running
into an unusual deadlocking problem.

For testing purposes, we're just running one web server (threaded
environment) against an index mounted on an NFS share. This machine performs
searches only against this index so it's not being touched. We have tried a
few different models so far:

1) Pooling IndexSearcher objects: Occasionally we would run into OutOfMemory
problems as we would not block if a request came through and all
IndexSearchers were already checked out, we would just create a temporary
one and then dispose of it once it was returned to the pool.

2) Create a new IndexSearcher each time: Every request to search would
create an IndexSearcher object. This quickly gave OutOfMemory errors, even
when we would close them out directly after.

3) Use a global IndexSearcher: This is the model we're working with now. The
model holds up fine under low-moderate load and is, in fact, much faster at
searching (probably due to some caching mechanism). Under heavy load though,
the CPU will spike up to 99% and never come back down until we kill -9 the
process. Also, as you ramp the load, we've discovered that search times go
up as well. Searches will generally come back after 40ms, but as the load
goes up the searches don't come back for up to 20 seconds.

We've been attempting to find where the problem is for the last week with no
luck. Our index is optimized, so there is only one file. Do we need to
synchronize access to the global IndexSearcher so that only one search can
run at a time? That poses a bit of a problem as if a particular search takes
a long time, all others will wait. This problem does not look like an
OutOrMemory error because the memory usage when the spike occurs is usually
in the range of 150meg used with a ceiling of 650meg. Anyone else
experiencing any problems like this or have any idea where we should be
looking? Thanks.




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message