lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: FileNotFoundException in ConcurrentMergeScheduler
Date Thu, 12 Jun 2008 10:39:15 GMT

Hi Grant,

My stress test is unable to reproduce this exception, either.  I'm  
adding Wikipedia docs to an index, using a high merge factor, then  
opening a new writer with low merge factor (5) and calling optimize.   
This forces concurrent merges to run during the optimize.

One more question: are your docs uniform?  Or, for example, do some  
of them have stored fields while others do not?

Mike

Grant Ingersoll wrote:

>
> On Jun 11, 2008, at 6:00 AM, Michael McCandless wrote:
>
>>
>> Grant Ingersoll wrote:
>>
>>>> Is more than one thread adding documents to the index?
>>>
>>> I don't believe so, but I am trying to reproduce.  I've only seen  
>>> it once, and don't have a lot of details, other than I noticed it  
>>> was on a specific file (.fdt) and was wondering if that was a  
>>> factor or not.  That is, maybe Paul could reproduce it.
>>
>> I think your exception differs from Paul's in an important way.   
>> Paul's exception means an entire segment (its CFS file) was  
>> deleted, which is very easily caused by accidentally allowing 2  
>> writers on the index at once.  But in your case, SegmentReader  
>> successfully opened the fnm file but then failed on the fdt, so,  
>> your segment wasn't deleted (at least not entirely).  So I think  
>> something different caused your exception.
>>
>>>> Any changes to the defaults in IndexWriter?
>>>
>>> It's the SolrIndexWriter.
>>
>> OK.  But what does your solrconfig.xml look like?
>
> It's pretty much the standard indexing setup that's on trunk.   
> Merge Factor 10, Max Buffered Docs 1000
>
>>
>>
>>>>
>>>>
>>>> After seeing that exception, does IndexReader.open also hit that  
>>>> exception (ie, is/was the index corrupt)?  Or does it only  
>>>> happen with BG merges?
>>>
>>> Not sure, unfortunately, I don't have a lot of info yet.  The  
>>> background exception happened during an optimize, if that matters  
>>> at all
>>
>> OK.  It'd be very useful to know if index was really corrupt  
>> (missing that segment) vs BG merge incorrectly, temporarily,  
>> thought it was supposed to merge that segment.
>>
>> Is this a largish index?
>
> couple million relatively small records
>
>>  Like, would there be so many segments that optimize would be  
>> running concurrent merges (> 2*mergeFactor segments)?  With  
>> ConcurrentMergeScheduler, optimize is now able to run multiple  
>> merges concurrently, if the index has enough segments.
>
> I don't know.  Like I said, the only reason I posted was b/c I  
> thought it sounded similar to Paul's.  I haven't been able to  
> reproduce it yet.
>
>>
>>
>> I'll run some stress tests, focusing on concurrency of merges  
>> during optimize...
>>
>> Which OS & JRE are you using?
>
> Linux, Java 1.6.0_05
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message