lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: FileNotFoundException in ConcurrentMergeScheduler
Date Tue, 10 Jun 2008 19:35:35 GMT

Grant,

Can you describe any details on how this app is using Lucene?  EG are  
you using autoCommit=false or true?  Is more than one thread adding  
documents to the index?  Any changes to the defaults in IndexWriter?

After seeing that exception, does IndexReader.open also hit that  
exception (ie, is/was the index corrupt)?  Or does it only happen  
with BG merges?

Mike

Grant Ingersoll wrote:

> Hi Paul,
>
> Not sure if this was resolved, but I don't think it was.  Can you  
> try reproducing this with setCompoundFile(false)?  That is, turn of  
> compound files.  I have an intermittent report of an exception that  
> looks eerily similar that I am trying to track down and I am not  
> using CFS and the exception is reporting the error in the .fdt  
> (stored fields info) file.  Unfortunately, I don't have any more  
> specifics at this time as it hasn't been reproduced since, but  
> thought that you might be able to get to it quicker since you seem  
> to be able to somewhat reproduce it.
>
> Exception is:
> java.io.FileNotFoundException: ..../index/_1l5.fdt (No such file or  
> directory)
> 	at java.io.RandomAccessFile.open(Native Method)
> 	at java.io.RandomAccessFile.<init>(Unknown Source)
> 	at org.apache.lucene.store.FSDirectory$FSIndexInput 
> $Descriptor.<init>(FSDirectory.java:506)
> 	at org.apache.lucene.store.FSDirectory$FSIndexInput.<init> 
> (FSDirectory.java:536)
> 	at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java: 
> 445)
> 	at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:75)
> 	at org.apache.lucene.index.SegmentReader.initialize 
> (SegmentReader.java:308)
> 	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
> 	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
> 	at org.apache.lucene.index.IndexWriter.mergeMiddle 
> (IndexWriter.java:3263)
> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2968)
> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run 
> (ConcurrentMergeScheduler.java:240)
>
> Of course, this may just be a red herring, since it is a background  
> thread running and it may just be a timing thing, but it might be  
> interesting if it consistently occurred in the same spot.
>
>
> Thanks,
> Grant
>
> On May 29, 2008, at 7:43 PM, Paul J. Lucas wrote:
>
>> I occasionally get a FileNotFoundException like:
>>
>> Exception in thread "Thread-44" org.apache.lucene.index.MergePolicy 
>> $MergeException: java.io.FileNotFoundException: /Stuff/Caches/ 
>> AuroraSupport/IM_IndexCache/INDEX/_27.cfs (No such file or directory)
>> 	at org.apache.lucene.index.ConcurrentMergeScheduler 
>> $MergeThread.run(ConcurrentMergeScheduler.java:271)
>> Caused by: java.io.FileNotFoundException: /Stuff/Caches/ 
>> AuroraSupport/IM_IndexCache/INDEX/_27.cfs (No such file or directory)
>> 	at java.io.RandomAccessFile.open(Native Method)
>> 	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
>> 	at org.apache.lucene.store.FSDirectory$FSIndexInput 
>> $Descriptor.<init>(FSDirectory.java:506)
>> 	at org.apache.lucene.store.FSDirectory$FSIndexInput.<init> 
>> (FSDirectory.java:536)
>> 	at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java: 
>> 445)
>> 	at org.apache.lucene.index.CompoundFileReader.<init> 
>> (CompoundFileReader.java:70)
>> 	at org.apache.lucene.index.SegmentReader.initialize 
>> (SegmentReader.java:277)
>> 	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
>> 	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
>> 	at org.apache.lucene.index.IndexWriter.mergeMiddle 
>> (IndexWriter.java:3263)
>> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2968)
>> 	at org.apache.lucene.index.ConcurrentMergeScheduler 
>> $MergeThread.run(ConcurrentMergeScheduler.java:240)
>>
>> I'm currently using the 2.3.2 version of Lucene.  The exception  
>> frequency has decreased since upgrading from 2.3.1.
>>
>> My code runs in a server that monitors the filesystem.  Whenever  
>> the contents of a directory change, my code unindexes all the  
>> files in the changed directory, then reindexes all the files in  
>> the directory.
>>
>> I'm not doing anything complicated in my code.  To create/open the  
>> index, I do:
>>
>> 	INDEX_DIR = new File( INDEX_CACHE_DIR, "INDEX" );
>> 	INDEX = FSDirectory.getDirectory( INDEX_DIR );
>> 	if ( IndexReader.isLocked( INDEX ) )
>> 	    IndexReader.unlock( INDEX );
>>
>> The isLocked()/unlock() is because sometimes the server process  
>> gets killed and leaves teh indexed locked.
>>
>> I have a thread than handles the unindexing/reindexing.  It gets  
>> changed from a BlockingQueue.  My unindex code is like:
>>
>> 	IndexWriter writer = new IndexWriter( INDEX, INDEX_ANALYZER,  
>> false );
>> 	final Term t = new Term( DIR_FIELD, dir.getAbsolutePath() );
>> 	writer.deleteDocuments( t );
>> 	writer.close();
>>
>> My indexing code is like:
>>
>> 	IndexWriter writer = new IndexWriter( INDEX, INDEX_ANALYZER );
>> 	writer.setMergeFactor( INDEX_MERGE_FACTOR );
>> 	Document doc = new Document();
>> 	// ... add fields to doc ...
>> 	writer.addDocument( doc );
>> 	writer.close();
>>
>> While that thread is executing, other threads can search the index  
>> by doing:
>>
>> 	IndexSearcher searcher = new IndexSearcher( INDEX );
>> 	// ... prepare Query and Sort ...
>> 	Hits hits = searcher.search( query, sort );
>> 	Iterator hitIterator  hits.iterator();
>> 	// ... iterate over hitIterator ...
>> 	searcher.close();
>>
>> Any ideas?  Is this a bug in Lucene?
>>
>> - Paul
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message