lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: No segment files found/ Searcher error
Date Wed, 05 Nov 2008 18:08:58 GMT

If you are able to leave the IndexSearcher open, and resubmit the  
query, and most likely it works the 2nd time around, it really seems  
like something intermittent is going wrong with your IO system.

That exception from CheckIndex is just happening when Lucene is trying  
to read bytes from the stored fields files.

Is there anything in the machine's system logs indicating problems?

What OS / filesystem is that again?

Mike

JulieSoko wrote:

>
> Yesterday, when typing a reply I noticed an error ... the  
> CheckIndex.check
> was not checking a good directory.. so the segments not found error  
> was not
> correct.  I apologize.  I made the correction and this is the  
> correct error
> that occurs when an i/o error is thrown.  I ran the CheckIndex.check  
> method
> after the i/o exception was caught:
>
> Segments file=segments_296 numSegments=1  
> version=FORMAT_SHARED_DOC_STORE
> [Lucene 2.3]
>     1 of 1: name=+1dq docCount=1788454
>     compund=true
>     numFiles=1
>     size (MB) = 331.125
>     no deletions
>     test: open reader....... OK
>     test: fields, norms...... OK [4 fields]
>     test: terms, freq, prox...2008-11-05 15:12:33,357 INFO Cache:195 -
> Purging 0 of 4 Cached results.
> 2008-11-05 15:12:33,358 INFO Cache: 196 - 956018816 of 1073676288  
> bytes of
> memory are currently free
> OK [3409802 terms; 19311204 terms/docs pairs; 19842525 tokens]
>    test: stored fields....... Wed Nov 05 15:12:59 UTC 2008: Purging  
> Query
> Cache 0 of 9 query cached results.
> Wed Nov 05 15:12:59 UTC 2008: 955521656 of 1073676288 bytes of  
> memory are
> currently free.
> FAILED
>    WARNING: would remove reference to this segment (-fix was not
> specified); full exception:
> java.io.IOException: Input/Output error
>   at java.io.RandomAccessfile.readBytes(Native Method)
>   at java.io.RandomAccessFile.read(RandomAccessfile.java:315)
>   at
> org.apache.lucene.store.FSDirectory 
> $FSIndexInput.readInternal(FSDirectory.java:550)
>   at
> org 
> .apache 
> .lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java: 
> 131)
>   at
> org.apache.lucene.index.CompoundFileReader 
> $CSIndexInput.readInternal(CompoundFileReader.java:240)
>  at
> org 
> .apache 
> .lucene.store.BufferedIndexInput.refill(bufferedIndexInput.java:152)
>  at
> org 
> .apache 
> .lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
>  at org.apache.lucene.store.IndexInput.readChars(IndexInput.java:126)
>  at org.apache.lucene.store.IndexInput.readString(IndexInput.java:112)
>  at org.apache.lucene.index.FieldsReader.addfield(FieldsReader.java: 
> 324)
>  at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:166)
>  at  
> org.apache.lucene.index.SegmentReader.document(SegmentReader.java:659)
>  at org.apache.lucene.index.IndexReader.document(IndexReader.java:525)
>  at org.apache.lucene.index.CheckIndex.check(checkIndex.java:234)
>  at
> gminer 
> .rest 
> .content 
> .RestContentSearchServlet.doGet(RestContentSearchServlet.java:269)
>  .................. continues on  and then
>
> WARNING: 1 broken segments detected
> WARNING: 1788454 documents would be lost if -fix were specified
>
> NOTE: would write new segments file [-fix was not specified]
>
> ************************************************
> Then I wrote out the return status from the check method call and it  
> was "
> false".
>
> The Normal Input/Output error follows...
>
> NOW, I am leaving the IndexSearcher open for the current day and if I
> resubmit the query... the query will most likely work the 2nd  
> time... Is
> there something to be done to the index so that there are not the  
> errors
> above?
>
> I appreciate all your help.
> Julie
>
>
>
>
>
>
>
>
>
> JulieSoko wrote:
>>
>> I am narrowing down this problem that I have had for a week  
>> now...   I am
>> using lucene version 2.3.1 and 64 bit java versio 1.5.0-12-b04  
>> running on
>> Linux box.  We are merging indexes every 60 seconds and there are  
>> 1..*
>> searches occuring at anytime on the indexes.  The problem is that  
>> we will
>> get an Input /Output error trying to read the index for a search
>> randomly... say every 5th search.  I have posted the error before,  
>> but
>> have narrowed it down, I believe, to a merge issue.
>>
>> This is the error that a searcher will output at random times:
>> java.io.IOException: Input/output error
>> java.io.RandomAccessFile.readBytes(Native Method)
>> java.io.RandomAccessFile.read(RandomAccessFile.java:315)
>> at
>> org.apache.lucene.store.FSDirecotry 
>> $FSIndexInput.readInternal(FSDirectory.java:550)
>> at
>> org 
>> .apache 
>> .lucene.store.BufferedIndexInput.readBytes(BufferedInputInput.java: 
>> 131)
>> at
>> org.apache.lucene.index.CompoundFileReader 
>> $CSIndexInput.readInternal(CompoundFileReader.java:240)
>> at
>> org 
>> .apache 
>> .lucene.instoreBufferedIndexInput.refill(BufferedIndexInput.java:
>> 152)
>> at
>> org 
>> .apache 
>> .lucene.instoreBufferedIndexInput.readByte(BufferedIndexInput.java:
>> 152)
>> at org.lucene.store.IndexInput.readVInt(IndexInput.java:76)
>> at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:63)
>> at  
>> org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java: 
>> 123)
>> at
>> org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java: 
>> 154)
>> at
>> org 
>> .apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java: 
>> 223)
>> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java: 
>> 217)
>> at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java: 
>> 678)
>> at  
>> org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:87)
>> at org.apache.lucene.search.Searcher.docFreqs(searcher.java:118)
>> at
>> org 
>> .apache.lucene.search.MultiSearcher.createWeight(MultiSearcher.java: 
>> 311)
>> at org.apache.lucene.search.Searcher.search(Searcher.java:178)
>>
>>
>> ******************************************************************************
>> NOW, when I get the above Exception, I check the index using the
>> CheckIndex.check method... As part of the check , this exception is
>> thrown:
>>
>>
>> Error: could not read any segments file in directory
>> java.io.FileNotFoundException: no segments* file found in
>> org.apache.lucen.store.FSDirectory@appfs/rt10/jetty/20081103
>>    at
>> org.apache.lucene.index.SegmentInfos 
>> $findSegmentsFile.run(SegementInfos.java:587)
>> .....
>>
>> Is there any point in the merging of indexes that the segment files  
>> are
>> removed?   If I rerun the search, right after this error occurs, the
>> search is ok... I do open a new IndexSearcher...
>>
>> The IndexWriter code is this:
>>   IndexWriter combinedWriter = new IndexWriter(currentMergeDir, new
>> StandardAnalyzer());
>>   combinedWriter.addIndexes(dirToMerge);
>>   combinedWriter.flush();
>>   combinedWriter.close();
>>
>>
>> As you can see above, each time there is a merge a new IndexWriter is
>> created, indexes added, flushed and closed.
>> I know you are not supposed to have synchronization issues between  
>> writing
>> and flushing , but could there be an issue when you are creating a  
>> new
>> searcher at the instant where the files are merged and there are no
>> segments in a dir???
>>
>> Thanks,
>> Julie
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> -- 
> View this message in context: http://www.nabble.com/No-segment-files-found--Searcher-error-tp20305354p20343211.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message