lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JulieSoko <julie.sokolow...@saic.com>
Subject Re: No segment files found/ Searcher error
Date Wed, 05 Nov 2008 15:36:31 GMT

Yesterday, when typing a reply I noticed an error ... the CheckIndex.check
was not checking a good directory.. so the segments not found error was not
correct.  I apologize.  I made the correction and this is the correct error
that occurs when an i/o error is thrown.  I ran the CheckIndex.check method
after the i/o exception was caught:

Segments file=segments_296 numSegments=1 version=FORMAT_SHARED_DOC_STORE
[Lucene 2.3]
     1 of 1: name=+1dq docCount=1788454
     compund=true
     numFiles=1
     size (MB) = 331.125
     no deletions
     test: open reader....... OK
     test: fields, norms...... OK [4 fields]
     test: terms, freq, prox...2008-11-05 15:12:33,357 INFO Cache:195 -
Purging 0 of 4 Cached results.
2008-11-05 15:12:33,358 INFO Cache: 196 - 956018816 of 1073676288 bytes of
memory are currently free
OK [3409802 terms; 19311204 terms/docs pairs; 19842525 tokens]
    test: stored fields....... Wed Nov 05 15:12:59 UTC 2008: Purging Query
Cache 0 of 9 query cached results.
Wed Nov 05 15:12:59 UTC 2008: 955521656 of 1073676288 bytes of memory are
currently free.
FAILED
    WARNING: would remove reference to this segment (-fix was not
specified); full exception:
java.io.IOException: Input/Output error
   at java.io.RandomAccessfile.readBytes(Native Method)
   at java.io.RandomAccessFile.read(RandomAccessfile.java:315)
   at
org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:550)
   at
org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:131)
   at
org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:240)
  at
org.apache.lucene.store.BufferedIndexInput.refill(bufferedIndexInput.java:152)
  at
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
  at org.apache.lucene.store.IndexInput.readChars(IndexInput.java:126)
  at org.apache.lucene.store.IndexInput.readString(IndexInput.java:112)
  at org.apache.lucene.index.FieldsReader.addfield(FieldsReader.java:324)
  at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:166)
  at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:659)
  at org.apache.lucene.index.IndexReader.document(IndexReader.java:525)
  at org.apache.lucene.index.CheckIndex.check(checkIndex.java:234)
  at
gminer.rest.content.RestContentSearchServlet.doGet(RestContentSearchServlet.java:269)
  .................. continues on  and then

WARNING: 1 broken segments detected
WARNING: 1788454 documents would be lost if -fix were specified

NOTE: would write new segments file [-fix was not specified]

************************************************
Then I wrote out the return status from the check method call and it was "
false".

The Normal Input/Output error follows... 

NOW, I am leaving the IndexSearcher open for the current day and if I
resubmit the query... the query will most likely work the 2nd time... Is
there something to be done to the index so that there are not the errors
above?  

I appreciate all your help.
Julie









JulieSoko wrote:
> 
> I am narrowing down this problem that I have had for a week now...   I am
> using lucene version 2.3.1 and 64 bit java versio 1.5.0-12-b04 running on 
> Linux box.  We are merging indexes every 60 seconds and there are 1..*
> searches occuring at anytime on the indexes.  The problem is that we will
> get an Input /Output error trying to read the index for a search
> randomly... say every 5th search.  I have posted the error before, but
> have narrowed it down, I believe, to a merge issue.  
> 
> This is the error that a searcher will output at random times:
>  java.io.IOException: Input/output error
>  java.io.RandomAccessFile.readBytes(Native Method)
>  java.io.RandomAccessFile.read(RandomAccessFile.java:315)
>  at
> org.apache.lucene.store.FSDirecotry$FSIndexInput.readInternal(FSDirectory.java:550)
>  at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedInputInput.java:131)
>  at
> org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:240)
> at
> org.apache.lucene.instoreBufferedIndexInput.refill(BufferedIndexInput.java:
> 152)
> at
> org.apache.lucene.instoreBufferedIndexInput.readByte(BufferedIndexInput.java:
> 152)
> at org.lucene.store.IndexInput.readVInt(IndexInput.java:76)
> at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:63)
> at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:123)
> at
> org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:154)
> at
> org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223)
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217)
> at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:678)
> at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:87)
> at org.apache.lucene.search.Searcher.docFreqs(searcher.java:118)
> at
> org.apache.lucene.search.MultiSearcher.createWeight(MultiSearcher.java:311)
> at org.apache.lucene.search.Searcher.search(Searcher.java:178) 
> 
> 
> ******************************************************************************
> NOW, when I get the above Exception, I check the index using the
> CheckIndex.check method... As part of the check , this exception is
> thrown:
> 
> 
> Error: could not read any segments file in directory
> java.io.FileNotFoundException: no segments* file found in
> org.apache.lucen.store.FSDirectory@appfs/rt10/jetty/20081103 
>     at
> org.apache.lucene.index.SegmentInfos$findSegmentsFile.run(SegementInfos.java:587)
>  .....
> 
> Is there any point in the merging of indexes that the segment files are
> removed?   If I rerun the search, right after this error occurs, the
> search is ok... I do open a new IndexSearcher...   
> 
> The IndexWriter code is this:
>    IndexWriter combinedWriter = new IndexWriter(currentMergeDir, new
> StandardAnalyzer());
>    combinedWriter.addIndexes(dirToMerge);
>    combinedWriter.flush();
>    combinedWriter.close();
> 
> 
> As you can see above, each time there is a merge a new IndexWriter is
> created, indexes added, flushed and closed.
> I know you are not supposed to have synchronization issues between writing
> and flushing , but could there be an issue when you are creating a new
> searcher at the instant where the files are merged and there are no
> segments in a dir???
> 
> Thanks,
> Julie
> 
> 
> 
> 
> 
> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/No-segment-files-found--Searcher-error-tp20305354p20343211.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message