lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Corrupted Indexes Under Lucene 2.3 (and 2.3.1)
Date Sat, 01 Mar 2008 10:15:50 GMT

Note that there are actually two concurrency issues to guard against
here:

   * Document itself cannot be changed (fields added or removed) from
     multiple threads without external synchronization.

   * Document cannot be changed from one thread while another thread is
     calling writer.addDocument.  It's this case that's causing your
     1st exception.

Please report back on whether adding correct external synchronization
resolves this!

I can explain why the original exception leads to index corruption:
IndexWriter incorrectly increments the buffered document count even
when there is an exception and the document was not in fact added to
the segment.  I will fix this so an exception when init'ing a document
doesn't then lead to index corruption...

Thanks for reporting this Tyler!

Mike

Yonik Seeley wrote:

> On Fri, Feb 29, 2008 at 7:05 PM, Tyler V <tylervsd@gmail.com> wrote:
>> Mike -- Thanks so much for the prompt reply.
>>
>>  You are right, we are accessing these documents with multiple  
>> threads
>>  (and have always been). However, I am wondering if the increased
>>  indexing speed in 2.3 has revealed a hidden concurrency issue.
>
> You are modifying the documents from multiple threads?
>
> My fault... I removed the synchronization on Document (changed from
> Vector to ArrayList).  It was never guaranteed to be thread-safe for
> modification, and almost never makes sense without external
> synchronization anyway.
>
> If you really need to modify a single document from multiple threads,
> please synchronize.
> That explains the first exception, but no the second.  I assume you
> aren't still changing the document while it's being indexed?
> It appears as if the original exception causes corruption.
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message