lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie <ja...@stimulussoft.com>
Subject Re: Lucene 2.3.1 IndexReader and IndexWriter Synchronization Issues
Date Sat, 29 Mar 2008 09:49:54 GMT
By the way, the line numbers in the last email apply to the SVN trunk 
code. I switch over to the latest code to see if this would make a 
difference. Alas, the problem still exists!

Jamie wrote:
> Hi Michael
>
> Glad you asked and thanks in advance for your help! I am trying to 
> reindex 325 emails. Its a pretty straight forward use of Lucene. I 
> create an IndexWriter, write a whole bunch of documents and then close 
> the Index every 2 seconds. See the attached source code. In case, you 
> are wondering there is only ever one VolumeIndex ever created. 
> Following, there is only ever one IndexWriter created in the entire 
> application. After further testing I dont think the problem is related 
> to IndexReader since the problem still occurs even if I dont execute a 
> search. The amount emails indexed varies between around 305 and 325. 
> The exception that gets thrown prevents the emails from being indexed. 
> I've tried  removing the synchronization and doing a simple:
>
> synchronized(this) {
>    openIndex()
>    write()
>    closeIndex()
> }
>
> i.e. open and close the index between every document write and the 
> problem still appears.
>
> For a recap, here is the exception:
>
> java.io.IOException: Cannot overwrite: C:\index9121\_1.cfs
>    at 
> org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:433)
>    at 
> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:150) 
>
>    at 
> org.apache.lucene.index.DocumentsWriter.createCompoundFile(DocumentsWriter.java:587)

>
>    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3251)
>    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3110)
>    at 
> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1659)
>    at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1633)
>    at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1601)
>
> I cleared out the logs and executed another reindex and the following 
> debug information was outputted by Lucene. Unfortunately, the debug 
> output does not include any information on the error. I am not sure 
> why this is the case, but I can assure you that the log information 
> was generated by the reindexing operation. Any ideas on what this 
> might be? Can you suggest a workaround?
>
> As an aside note, is Lucene 2.2.0 compatible with Lucene 2.3.0 
> indexes? If I cant sort this out in the next couple of days, I may 
> need to switch everyone back to Lucene 2.2.0 temporarily until this 
> problem is resolved.
>
> IFD [http-8090-2]: setInfoStream 
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@1ebf305 
>
> IW 22 [http-8090-2]: setInfoStream: 
> dir=org.apache.lucene.store.FSDirectory@C:\index9121 autoCommit=false 
> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@19ffd6f 
> mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@b799cf 
> ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 
> maxFieldLength=50000 index=_0:C54 _1:C82 _2:C14 _3:c36 _4:c6 _5:c52 
> _6:c53
> IW 22 [Timer-0]: now flush at close
> IW 22 [Timer-0]:   flush: segment=_7 docStoreSegment=_7 
> docStoreOffset=0 flushDocs=true flushDeletes=true flushDocStores=true 
> numDocs=28 numBufDelTerms=0
> IW 22 [Timer-0]:   index before flush _0:C54 _1:C82 _2:C14 _3:c36 
> _4:c6 _5:c52 _6:c53
> IW 22 [Timer-0]: DW: flush postings as segment _7 numDocs=28
> IW 22 [Timer-0]: DW: closeDocStore: 2 files to flush to segment _7 
> numDocs=28
> IW 22 [Timer-0]: DW:   oldRAMSize=245760 newFlushedSize=38119 
> docs/MB=770.223 new/old=15.511%
> IFD [Timer-0]: now checkpoint "segments_8" [8 segments ; isCommit = 
> false]
> IFD [Timer-0]: delete pending file _f.cfs
> IFD [Timer-0]: delete "_f.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _h.cfs
> IFD [Timer-0]: delete "_h.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _d.cfs
> IFD [Timer-0]: delete "_d.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _c.cfs
> IFD [Timer-0]: delete "_c.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _b.cfs
> IFD [Timer-0]: delete "_b.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _g.cfs
> IFD [Timer-0]: delete "_g.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _e.cfs
> IFD [Timer-0]: delete "_e.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try 
> later.
> IFD [Timer-0]: now checkpoint "segments_8" [8 segments ; isCommit = 
> false]
> IFD [Timer-0]: delete pending file _f.cfs
> IFD [Timer-0]: delete "_f.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _h.cfs
> IFD [Timer-0]: delete "_h.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _d.cfs
> IFD [Timer-0]: delete "_d.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _c.cfs
> IFD [Timer-0]: delete "_c.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _b.cfs
> IFD [Timer-0]: delete "_b.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _g.cfs
> IFD [Timer-0]: delete "_g.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _e.cfs
> IFD [Timer-0]: delete "_e.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete "_7.fnm"
> IFD [Timer-0]: delete "_7.frq"
> IFD [Timer-0]: delete "_7.prx"
> IFD [Timer-0]: delete "_7.tis"
> IFD [Timer-0]: delete "_7.tii"
> IFD [Timer-0]: delete "_7.nrm"
> IFD [Timer-0]: delete "_7.fdx"
> IFD [Timer-0]: delete "_7.fdt"
> IW 22 [Timer-0]: LMP: findMerges: 8 segments
> IW 22 [Timer-0]: LMP:   level -1.0 to 5.202747: 8 segments
> IW 22 [Timer-0]: CMS: now merge
> IW 22 [Timer-0]: CMS:   index: _0:C54 _1:C82 _2:C14 _3:c36 _4:c6 
> _5:c52 _6:c53 _7:c28
> IW 22 [Timer-0]: CMS:   no more merges pending; now return
> IW 22 [Timer-0]: now call final commit()
> IW 22 [Timer-0]: start commit() skipWait=true sizeInBytes=0
> IW 22 [Timer-0]: commit index=_0:C54 _1:C82 _2:C14 _3:c36 _4:c6 _5:c52 
> _6:c53 _7:c28
> IW 22 [Timer-0]: now sync _7.cfs
> IW 22 [Timer-0]: commit complete
> IFD [Timer-0]: now checkpoint "segments_9" [8 segments ; isCommit = true]
> IFD [Timer-0]: delete pending file _f.cfs
> IFD [Timer-0]: delete "_f.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _h.cfs
> IFD [Timer-0]: delete "_h.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _d.cfs
> IFD [Timer-0]: delete "_d.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _c.cfs
> IFD [Timer-0]: delete "_c.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _b.cfs
> IFD [Timer-0]: delete "_b.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _g.cfs
> IFD [Timer-0]: delete "_g.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _e.cfs
> IFD [Timer-0]: delete "_e.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try 
> later.
> IFD [Timer-0]: deleteCommits: now decRef commit "segments_8"
> IFD [Timer-0]: delete "segments_8"
> IW 22 [Timer-0]: done all syncs
> IW 22 [Timer-0]: at close: _0:C54 _1:C82 _2:C14 _3:c36 _4:c6 _5:c52 
> _6:c53 _7:c28
> IFD [Timer-0]: delete pending file _f.cfs
> IFD [Timer-0]: delete "_f.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _h.cfs
> IFD [Timer-0]: delete "_h.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _d.cfs
> IFD [Timer-0]: delete "_d.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _c.cfs
> IFD [Timer-0]: delete "_c.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _b.cfs
> IFD [Timer-0]: delete "_b.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _g.cfs
> IFD [Timer-0]: delete "_g.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try 
> later.
> IFD [Timer-0]: delete pending file _e.cfs
> IFD [Timer-0]: delete "_e.cfs"
> IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs": 
> java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try 
> later.
>
>
> Michael McCandless wrote:
>>
>> That use case of Lucene should be fine, ie no further synchronization 
>> should be necessary.
>>
>> Your debug output below is great, but it doesn't seem to cover the 
>> occurrence of that exception.  Can you post the full debug output?
>>
>> Can you describe how you reindex in more detail?  Are you manually 
>> removing files from the index and then opening a new IndexWriter with 
>> create=true?
>>
>> Mike
>
> ------------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


-- 
Stimulus Software - MailArchiva
Email Archiving And Compliance
USA Tel: +1-713-366-8072 ext 3
UK Tel: +44-20-80991035 ext 3
Email: jamie@stimulussoft.com
Web: http://www.mailarchiva.com

To receive MailArchiva Enterprise Edition product announcements, send a message to: <mailarchiva-enterprise-edition-subscribe@stimulussoft.com>



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message