lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <gsing...@apache.org>
Subject Re: FileNotFoundException in ConcurrentMergeScheduler
Date Wed, 11 Jun 2008 11:28:24 GMT

On Jun 11, 2008, at 6:00 AM, Michael McCandless wrote:

>
> Grant Ingersoll wrote:
>
>>> Is more than one thread adding documents to the index?
>>
>> I don't believe so, but I am trying to reproduce.  I've only seen  
>> it once, and don't have a lot of details, other than I noticed it  
>> was on a specific file (.fdt) and was wondering if that was a  
>> factor or not.  That is, maybe Paul could reproduce it.
>
> I think your exception differs from Paul's in an important way.   
> Paul's exception means an entire segment (its CFS file) was deleted,  
> which is very easily caused by accidentally allowing 2 writers on  
> the index at once.  But in your case, SegmentReader successfully  
> opened the fnm file but then failed on the fdt, so, your segment  
> wasn't deleted (at least not entirely).  So I think something  
> different caused your exception.
>
>>> Any changes to the defaults in IndexWriter?
>>
>> It's the SolrIndexWriter.
>
> OK.  But what does your solrconfig.xml look like?

It's pretty much the standard indexing setup that's on trunk.  Merge  
Factor 10, Max Buffered Docs 1000

>
>
>>>
>>>
>>> After seeing that exception, does IndexReader.open also hit that  
>>> exception (ie, is/was the index corrupt)?  Or does it only happen  
>>> with BG merges?
>>
>> Not sure, unfortunately, I don't have a lot of info yet.  The  
>> background exception happened during an optimize, if that matters  
>> at all
>
> OK.  It'd be very useful to know if index was really corrupt  
> (missing that segment) vs BG merge incorrectly, temporarily, thought  
> it was supposed to merge that segment.
>
> Is this a largish index?

couple million relatively small records

>  Like, would there be so many segments that optimize would be  
> running concurrent merges (> 2*mergeFactor segments)?  With  
> ConcurrentMergeScheduler, optimize is now able to run multiple  
> merges concurrently, if the index has enough segments.

I don't know.  Like I said, the only reason I posted was b/c I thought  
it sounded similar to Paul's.  I haven't been able to reproduce it yet.

>
>
> I'll run some stress tests, focusing on concurrency of merges during  
> optimize...
>
> Which OS & JRE are you using?

Linux, Java 1.6.0_05



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message