lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie <ja...@stimulussoft.com>
Subject Re: Lucene 2.3.1 IndexReader and IndexWriter Synchronization Issues
Date Sat, 29 Mar 2008 09:40:54 GMT
Hi Michael

Glad you asked and thanks in advance for your help! I am trying to 
reindex 325 emails. Its a pretty straight forward use of Lucene. I 
create an IndexWriter, write a whole bunch of documents and then close 
the Index every 2 seconds. See the attached source code. In case, you 
are wondering there is only ever one VolumeIndex ever created. 
Following, there is only ever one IndexWriter created in the entire 
application. After further testing I dont think the problem is related 
to IndexReader since the problem still occurs even if I dont execute a 
search. The amount emails indexed varies between around 305 and 325. The 
exception that gets thrown prevents the emails from being indexed. I've 
tried  removing the synchronization and doing a simple:

synchronized(this) {
    openIndex()
    write()
    closeIndex()
}

i.e. open and close the index between every document write and the 
problem still appears.

For a recap, here is the exception:

java.io.IOException: Cannot overwrite: C:\index9121\_1.cfs
    at 
org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:433)
    at 
org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:150)
    at 
org.apache.lucene.index.DocumentsWriter.createCompoundFile(DocumentsWriter.java:587)
    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3251)
    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3110)
    at 
org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1659)
    at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1633)
    at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1601)

I cleared out the logs and executed another reindex and the following 
debug information was outputted by Lucene. Unfortunately, the debug 
output does not include any information on the error. I am not sure why 
this is the case, but I can assure you that the log information was 
generated by the reindexing operation. Any ideas on what this might be? 
Can you suggest a workaround?

As an aside note, is Lucene 2.2.0 compatible with Lucene 2.3.0 indexes? 
If I cant sort this out in the next couple of days, I may need to switch 
everyone back to Lucene 2.2.0 temporarily until this problem is resolved.

IFD [http-8090-2]: setInfoStream 
deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@1ebf305
IW 22 [http-8090-2]: setInfoStream: 
dir=org.apache.lucene.store.FSDirectory@C:\index9121 autoCommit=false 
mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@19ffd6f 
mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@b799cf 
ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 
maxFieldLength=50000 index=_0:C54 _1:C82 _2:C14 _3:c36 _4:c6 _5:c52 _6:c53
IW 22 [Timer-0]: now flush at close
IW 22 [Timer-0]:   flush: segment=_7 docStoreSegment=_7 docStoreOffset=0 
flushDocs=true flushDeletes=true flushDocStores=true numDocs=28 
numBufDelTerms=0
IW 22 [Timer-0]:   index before flush _0:C54 _1:C82 _2:C14 _3:c36 _4:c6 
_5:c52 _6:c53
IW 22 [Timer-0]: DW: flush postings as segment _7 numDocs=28
IW 22 [Timer-0]: DW: closeDocStore: 2 files to flush to segment _7 
numDocs=28
IW 22 [Timer-0]: DW:   oldRAMSize=245760 newFlushedSize=38119 
docs/MB=770.223 new/old=15.511%
IFD [Timer-0]: now checkpoint "segments_8" [8 segments ; isCommit = false]
IFD [Timer-0]: delete pending file _f.cfs
IFD [Timer-0]: delete "_f.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs": 
java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _h.cfs
IFD [Timer-0]: delete "_h.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs": 
java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _d.cfs
IFD [Timer-0]: delete "_d.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs": 
java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _c.cfs
IFD [Timer-0]: delete "_c.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs": 
java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _b.cfs
IFD [Timer-0]: delete "_b.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs": 
java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _g.cfs
IFD [Timer-0]: delete "_g.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs": 
java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _e.cfs
IFD [Timer-0]: delete "_e.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs": 
java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try later.
IFD [Timer-0]: now checkpoint "segments_8" [8 segments ; isCommit = false]
IFD [Timer-0]: delete pending file _f.cfs
IFD [Timer-0]: delete "_f.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs": 
java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _h.cfs
IFD [Timer-0]: delete "_h.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs": 
java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _d.cfs
IFD [Timer-0]: delete "_d.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs": 
java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _c.cfs
IFD [Timer-0]: delete "_c.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs": 
java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _b.cfs
IFD [Timer-0]: delete "_b.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs": 
java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _g.cfs
IFD [Timer-0]: delete "_g.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs": 
java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _e.cfs
IFD [Timer-0]: delete "_e.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs": 
java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try later.
IFD [Timer-0]: delete "_7.fnm"
IFD [Timer-0]: delete "_7.frq"
IFD [Timer-0]: delete "_7.prx"
IFD [Timer-0]: delete "_7.tis"
IFD [Timer-0]: delete "_7.tii"
IFD [Timer-0]: delete "_7.nrm"
IFD [Timer-0]: delete "_7.fdx"
IFD [Timer-0]: delete "_7.fdt"
IW 22 [Timer-0]: LMP: findMerges: 8 segments
IW 22 [Timer-0]: LMP:   level -1.0 to 5.202747: 8 segments
IW 22 [Timer-0]: CMS: now merge
IW 22 [Timer-0]: CMS:   index: _0:C54 _1:C82 _2:C14 _3:c36 _4:c6 _5:c52 
_6:c53 _7:c28
IW 22 [Timer-0]: CMS:   no more merges pending; now return
IW 22 [Timer-0]: now call final commit()
IW 22 [Timer-0]: start commit() skipWait=true sizeInBytes=0
IW 22 [Timer-0]: commit index=_0:C54 _1:C82 _2:C14 _3:c36 _4:c6 _5:c52 
_6:c53 _7:c28
IW 22 [Timer-0]: now sync _7.cfs
IW 22 [Timer-0]: commit complete
IFD [Timer-0]: now checkpoint "segments_9" [8 segments ; isCommit = true]
IFD [Timer-0]: delete pending file _f.cfs
IFD [Timer-0]: delete "_f.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs": 
java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _h.cfs
IFD [Timer-0]: delete "_h.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs": 
java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _d.cfs
IFD [Timer-0]: delete "_d.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs": 
java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _c.cfs
IFD [Timer-0]: delete "_c.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs": 
java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _b.cfs
IFD [Timer-0]: delete "_b.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs": 
java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _g.cfs
IFD [Timer-0]: delete "_g.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs": 
java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _e.cfs
IFD [Timer-0]: delete "_e.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs": 
java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try later.
IFD [Timer-0]: deleteCommits: now decRef commit "segments_8"
IFD [Timer-0]: delete "segments_8"
IW 22 [Timer-0]: done all syncs
IW 22 [Timer-0]: at close: _0:C54 _1:C82 _2:C14 _3:c36 _4:c6 _5:c52 
_6:c53 _7:c28
IFD [Timer-0]: delete pending file _f.cfs
IFD [Timer-0]: delete "_f.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs": 
java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _h.cfs
IFD [Timer-0]: delete "_h.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs": 
java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _d.cfs
IFD [Timer-0]: delete "_d.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs": 
java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _c.cfs
IFD [Timer-0]: delete "_c.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs": 
java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _b.cfs
IFD [Timer-0]: delete "_b.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs": 
java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _g.cfs
IFD [Timer-0]: delete "_g.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs": 
java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try later.
IFD [Timer-0]: delete pending file _e.cfs
IFD [Timer-0]: delete "_e.cfs"
IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs": 
java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try later.


Michael McCandless wrote:
>
> That use case of Lucene should be fine, ie no further synchronization 
> should be necessary.
>
> Your debug output below is great, but it doesn't seem to cover the 
> occurrence of that exception.  Can you post the full debug output?
>
> Can you describe how you reindex in more detail?  Are you manually 
> removing files from the index and then opening a new IndexWriter with 
> create=true?
>
> Mike


Mime
View raw message