lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Index corruption using Lucene 2.4.1 - thread safety issue?
Date Fri, 22 Jan 2010 08:14:21 GMT
On Thu, Jan 21, 2010 at 4:46 PM, Frank Geary <fgeary@acquiremedia.com> wrote:

> Nope.  When it's time to inactivate a RAMDir indexWriter, I get that
> directory,  close that writer, then clear out the directory.  Then after
> clearing out the directory, I create a new IW passing in the directory that
> was used previously.  No, I do not override the LockFactory - not familiar
> with it.

Hmm, curious.

> Hmm...for what it's worth we are using ConcurrentMergeScheduler.
> We do actaully have setInfoStream turned on for all our indexers.  I'll have
> to hunt down one of our logs that corresponds to one of the Exceptions
> though.  Thanks for that.

That could be helpful... eg it'd let us confirm every open IW is
closed before a new one is opened, on the same IW (assuming we get the
full output from all opened IWs).

> We delete the files explicitly, because what we found was that under high
> index loads (i.e. rapid rolling of RAMDir indexWriters) leaving the RAMDir
> around lead to garbage collection issues as old RAMDirs accumulated waiting
> for garbage collection to take place.  It turned out that deleting the files
> explicitly avoided those garbage collection problems.

That's very odd -- RAMDir should be "easy" to GC once it's closed.  It
(and RAMFile) have no finalizers.

Were you calling RAMDir.close()?  That clears its fileMap, which will
make all RAM consumed GCable.

The fact that you hit GC issues makes me suspect something, somewhere
is still hanging on to the RAMDir (ie, the IW isn't always being
closed).

If you go back to using a new RAMDir instance per IW, does the problem go away?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message