lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: readModifiedUTF8String stuck
Date Fri, 17 Apr 2009 09:38:25 GMT
On Fri, Apr 17, 2009 at 5:05 AM, MakMak <powgri@gmail.com> wrote:

> I am not retrieving many docs, the problem is that the whole file is stored
> in the doc. I need the file content for highlighter to work. But the files
> are normal-sized text files which in any case should not exceed 10-15mb.
> Retrieving 25 of them(page size), worst case scenario will take 250mb of
> data transfer which should not take more than 600 secs.

Maybe you can add some logging as to which docs each thread is
attempting to load.  Eg is it possible you accidentally have an
immense doc in the index?

You could also make a standalone tool that simply requests each doc
from the index and prints out its size.

Also, maybe run CheckIndex on your index just to make sure all is good.

> This may not be easily reproduced, but is easy to describe, we bombard our
> application with about 3-4 requests per second for 2 hours as a part of
> stress tests. Previously when we were on 2.3.2, these used to run just fine.
> We had to upgrade due to a memory leak issue for which CloseableThreadLocal
> was implemented in 2.4.1.

OK.

> The testing team did not reindex with the new lucene version, but they dont
> have to, right?

Right, it's fully back compatible, though you should get better
performance by reindexing.

> thats when they saw these stuck threads. The tests ran fine
> for an hour when responses were being returned timely. But after an hour,
> the threads just started to accumulate. I am not sure amount of RAM consumed
> during searches, will find that out.

Hmm, that's spooky.  Do the queries repeat at <= 1 hour during the
test?  Or is it possible in that 2nd hour of testing, a new query
appeared that hit a horribly immense document?

If they do repeat... maybe some sort of crazy GC is kicking in?
You're on a 64 bit JRE right?  How much heap do you give to it?

> Hope I have been clear, do you smell something fishy in this?

The fact that you only hit this new issue on upgrading to 2.4.1 is
certainly fishy.  Nothing else changed in the upgrade?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message