Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-Id: <F660481E-891F-4D92-A115-C6AC21AB61B7@apache.org>
From: Grant Ingersoll <gsingers@apache.org>
To: java-user@lucene.apache.org
In-Reply-To: <00C66825-EB8B-4B08-82B1-72CF53E25C33@mikemccandless.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v924)
Subject: Re: FileNotFoundException in ConcurrentMergeScheduler
Date: Wed, 11 Jun 2008 07:28:24 -0400
References: <EEBD1ADA-C2E8-41D1-B8AE-E90DFDC090B4@mac.com>
 <2CF563C9-66FC-424E-9069-AFDC7307A856@apache.org>
 <3554C441-6C62-4E34-BDB9-0606D99EEA6F@mikemccandless.com>
 <6E16D593-87FE-43A4-AC5F-5E1BED594B2B@apache.org>
 <00C66825-EB8B-4B08-82B1-72CF53E25C33@mikemccandless.com>


On Jun 11, 2008, at 6:00 AM, Michael McCandless wrote:

>
> Grant Ingersoll wrote:
>
>>> Is more than one thread adding documents to the index?
>>
>> I don't believe so, but I am trying to reproduce.  I've only seen  
>> it once, and don't have a lot of details, other than I noticed it  
>> was on a specific file (.fdt) and was wondering if that was a  
>> factor or not.  That is, maybe Paul could reproduce it.
>
> I think your exception differs from Paul's in an important way.   
> Paul's exception means an entire segment (its CFS file) was deleted,  
> which is very easily caused by accidentally allowing 2 writers on  
> the index at once.  But in your case, SegmentReader successfully  
> opened the fnm file but then failed on the fdt, so, your segment  
> wasn't deleted (at least not entirely).  So I think something  
> different caused your exception.
>
>>> Any changes to the defaults in IndexWriter?
>>
>> It's the SolrIndexWriter.
>
> OK.  But what does your solrconfig.xml look like?

It's pretty much the standard indexing setup that's on trunk.  Merge  
Factor 10, Max Buffered Docs 1000

>
>
>>>
>>>
>>> After seeing that exception, does IndexReader.open also hit that  
>>> exception (ie, is/was the index corrupt)?  Or does it only happen  
>>> with BG merges?
>>
>> Not sure, unfortunately, I don't have a lot of info yet.  The  
>> background exception happened during an optimize, if that matters  
>> at all
>
> OK.  It'd be very useful to know if index was really corrupt  
> (missing that segment) vs BG merge incorrectly, temporarily, thought  
> it was supposed to merge that segment.
>
> Is this a largish index?

couple million relatively small records

>  Like, would there be so many segments that optimize would be  
> running concurrent merges (> 2*mergeFactor segments)?  With  
> ConcurrentMergeScheduler, optimize is now able to run multiple  
> merges concurrently, if the index has enough segments.

I don't know.  Like I said, the only reason I posted was b/c I thought  
it sounded similar to Paul's.  I haven't been able to reproduce it yet.

>
>
> I'll run some stress tests, focusing on concurrency of merges during  
> optimize...
>
> Which OS & JRE are you using?

Linux, Java 1.6.0_05


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org