lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie <ja...@stimulussoft.com>
Subject Re: Lucene 2.3.1 IndexReader and IndexWriter Synchronization Issues
Date Sat, 29 Mar 2008 11:00:40 GMT
H Michael

Well spotted! Thanks for the tip and your precious time.

Thanks!

Jamie


Michael McCandless wrote:
> OK indeed from your code you are manually removing files from the
> index (the "deleteIndex" method).  You shouldn't do that.
>
> I think what's happening is deleteIndex is called when an IndexReader
> has the index open (do you see those "failed to delete file in
> existing index" log messages?), then you open a writer on that
> directory.  That would exactly explain your exception.
>
> The fix is simple: never remove files from the index yourself.  Let
> IndexWriter do it, by opening with create=true.  It will take care of
> removing the old files, retrying over time, etc.
>
> And, most importantly, this allows you to create a new index in a
> directory even while an IndexReader is still using it.  The reader
> will be fine, and then when it reopens it will switch to the new
> index.
>
> Lucene 2.3 can read 2.2 indices but not vice versa, so you can't
> downgrade unless you completely remove your index.
>
> Mike
>
> Jamie <jamie@stimulussoft.com> wrote:
>   
>> By the way, the line numbers in the last email apply to the SVN trunk
>>  code. I switch over to the latest code to see if this would make a
>>  difference. Alas, the problem still exists!
>>
>>
>>
>>  Jamie wrote:
>>  > Hi Michael
>>  >
>>  > Glad you asked and thanks in advance for your help! I am trying to
>>  > reindex 325 emails. Its a pretty straight forward use of Lucene. I
>>  > create an IndexWriter, write a whole bunch of documents and then close
>>  > the Index every 2 seconds. See the attached source code. In case, you
>>  > are wondering there is only ever one VolumeIndex ever created.
>>  > Following, there is only ever one IndexWriter created in the entire
>>  > application. After further testing I dont think the problem is related
>>  > to IndexReader since the problem still occurs even if I dont execute a
>>  > search. The amount emails indexed varies between around 305 and 325.
>>  > The exception that gets thrown prevents the emails from being indexed.
>>  > I've tried  removing the synchronization and doing a simple:
>>  >
>>  > synchronized(this) {
>>  >    openIndex()
>>  >    write()
>>  >    closeIndex()
>>  > }
>>  >
>>  > i.e. open and close the index between every document write and the
>>  > problem still appears.
>>  >
>>  > For a recap, here is the exception:
>>  >
>>  > java.io.IOException: Cannot overwrite: C:\index9121\_1.cfs
>>  >    at
>>  > org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:433)
>>  >    at
>>  > org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:150)
>>  >
>>  >    at
>>  > org.apache.lucene.index.DocumentsWriter.createCompoundFile(DocumentsWriter.java:587)
>>  >
>>  >    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3251)
>>  >    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3110)
>>  >    at
>>  > org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1659)
>>  >    at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1633)
>>  >    at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1601)
>>  >
>>  > I cleared out the logs and executed another reindex and the following
>>  > debug information was outputted by Lucene. Unfortunately, the debug
>>  > output does not include any information on the error. I am not sure
>>  > why this is the case, but I can assure you that the log information
>>  > was generated by the reindexing operation. Any ideas on what this
>>  > might be? Can you suggest a workaround?
>>  >
>>  > As an aside note, is Lucene 2.2.0 compatible with Lucene 2.3.0
>>  > indexes? If I cant sort this out in the next couple of days, I may
>>  > need to switch everyone back to Lucene 2.2.0 temporarily until this
>>  > problem is resolved.
>>  >
>>  > IFD [http-8090-2]: setInfoStream
>>  > deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@1ebf305
>>  >
>>  > IW 22 [http-8090-2]: setInfoStream:
>>  > dir=org.apache.lucene.store.FSDirectory@C:\index9121 autoCommit=false
>>  > mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@19ffd6f
>>  > mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@b799cf
>>  > ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
>>  > maxFieldLength=50000 index=_0:C54 _1:C82 _2:C14 _3:c36 _4:c6 _5:c52
>>  > _6:c53
>>  > IW 22 [Timer-0]: now flush at close
>>  > IW 22 [Timer-0]:   flush: segment=_7 docStoreSegment=_7
>>  > docStoreOffset=0 flushDocs=true flushDeletes=true flushDocStores=true
>>  > numDocs=28 numBufDelTerms=0
>>  > IW 22 [Timer-0]:   index before flush _0:C54 _1:C82 _2:C14 _3:c36
>>  > _4:c6 _5:c52 _6:c53
>>  > IW 22 [Timer-0]: DW: flush postings as segment _7 numDocs=28
>>  > IW 22 [Timer-0]: DW: closeDocStore: 2 files to flush to segment _7
>>  > numDocs=28
>>  > IW 22 [Timer-0]: DW:   oldRAMSize=245760 newFlushedSize=38119
>>  > docs/MB=770.223 new/old=15.511%
>>  > IFD [Timer-0]: now checkpoint "segments_8" [8 segments ; isCommit =
>>  > false]
>>  > IFD [Timer-0]: delete pending file _f.cfs
>>  > IFD [Timer-0]: delete "_f.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _h.cfs
>>  > IFD [Timer-0]: delete "_h.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _d.cfs
>>  > IFD [Timer-0]: delete "_d.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _c.cfs
>>  > IFD [Timer-0]: delete "_c.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _b.cfs
>>  > IFD [Timer-0]: delete "_b.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _g.cfs
>>  > IFD [Timer-0]: delete "_g.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _e.cfs
>>  > IFD [Timer-0]: delete "_e.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: now checkpoint "segments_8" [8 segments ; isCommit =
>>  > false]
>>  > IFD [Timer-0]: delete pending file _f.cfs
>>  > IFD [Timer-0]: delete "_f.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _h.cfs
>>  > IFD [Timer-0]: delete "_h.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _d.cfs
>>  > IFD [Timer-0]: delete "_d.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _c.cfs
>>  > IFD [Timer-0]: delete "_c.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _b.cfs
>>  > IFD [Timer-0]: delete "_b.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _g.cfs
>>  > IFD [Timer-0]: delete "_g.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _e.cfs
>>  > IFD [Timer-0]: delete "_e.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete "_7.fnm"
>>  > IFD [Timer-0]: delete "_7.frq"
>>  > IFD [Timer-0]: delete "_7.prx"
>>  > IFD [Timer-0]: delete "_7.tis"
>>  > IFD [Timer-0]: delete "_7.tii"
>>  > IFD [Timer-0]: delete "_7.nrm"
>>  > IFD [Timer-0]: delete "_7.fdx"
>>  > IFD [Timer-0]: delete "_7.fdt"
>>  > IW 22 [Timer-0]: LMP: findMerges: 8 segments
>>  > IW 22 [Timer-0]: LMP:   level -1.0 to 5.202747: 8 segments
>>  > IW 22 [Timer-0]: CMS: now merge
>>  > IW 22 [Timer-0]: CMS:   index: _0:C54 _1:C82 _2:C14 _3:c36 _4:c6
>>  > _5:c52 _6:c53 _7:c28
>>  > IW 22 [Timer-0]: CMS:   no more merges pending; now return
>>  > IW 22 [Timer-0]: now call final commit()
>>  > IW 22 [Timer-0]: start commit() skipWait=true sizeInBytes=0
>>  > IW 22 [Timer-0]: commit index=_0:C54 _1:C82 _2:C14 _3:c36 _4:c6 _5:c52
>>  > _6:c53 _7:c28
>>  > IW 22 [Timer-0]: now sync _7.cfs
>>  > IW 22 [Timer-0]: commit complete
>>  > IFD [Timer-0]: now checkpoint "segments_9" [8 segments ; isCommit = true]
>>  > IFD [Timer-0]: delete pending file _f.cfs
>>  > IFD [Timer-0]: delete "_f.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _h.cfs
>>  > IFD [Timer-0]: delete "_h.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _d.cfs
>>  > IFD [Timer-0]: delete "_d.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _c.cfs
>>  > IFD [Timer-0]: delete "_c.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _b.cfs
>>  > IFD [Timer-0]: delete "_b.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _g.cfs
>>  > IFD [Timer-0]: delete "_g.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _e.cfs
>>  > IFD [Timer-0]: delete "_e.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: deleteCommits: now decRef commit "segments_8"
>>  > IFD [Timer-0]: delete "segments_8"
>>  > IW 22 [Timer-0]: done all syncs
>>  > IW 22 [Timer-0]: at close: _0:C54 _1:C82 _2:C14 _3:c36 _4:c6 _5:c52
>>  > _6:c53 _7:c28
>>  > IFD [Timer-0]: delete pending file _f.cfs
>>  > IFD [Timer-0]: delete "_f.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _h.cfs
>>  > IFD [Timer-0]: delete "_h.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _d.cfs
>>  > IFD [Timer-0]: delete "_d.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _c.cfs
>>  > IFD [Timer-0]: delete "_c.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _b.cfs
>>  > IFD [Timer-0]: delete "_b.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _g.cfs
>>  > IFD [Timer-0]: delete "_g.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try
>>  > later.
>>  > IFD [Timer-0]: delete pending file _e.cfs
>>  > IFD [Timer-0]: delete "_e.cfs"
>>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs":
>>  > java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try
>>  > later.
>>  >
>>  >
>>  > Michael McCandless wrote:
>>  >>
>>  >> That use case of Lucene should be fine, ie no further synchronization
>>  >> should be necessary.
>>  >>
>>  >> Your debug output below is great, but it doesn't seem to cover the
>>  >> occurrence of that exception.  Can you post the full debug output?
>>  >>
>>  >> Can you describe how you reindex in more detail?  Are you manually
>>  >> removing files from the index and then opening a new IndexWriter with
>>  >> create=true?
>>  >>
>>  >> Mike
>>  >
>>  > ------------------------------------------------------------------------
>>
>>     
>>  > ---------------------------------------------------------------------
>>  > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>  > For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>  --
>>  Stimulus Software - MailArchiva
>>  Email Archiving And Compliance
>>  USA Tel: +1-713-366-8072 ext 3
>>  UK Tel: +44-20-80991035 ext 3
>>  Email: jamie@stimulussoft.com
>>  Web: http://www.mailarchiva.com
>>
>>  To receive MailArchiva Enterprise Edition product announcements, send a message
to: <mailarchiva-enterprise-edition-subscribe@stimulussoft.com>
>>
>>
>>
>>
>>  ---------------------------------------------------------------------
>>  To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>  For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>     
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>   


-- 
Stimulus Software - MailArchiva
Email Archiving And Compliance
USA Tel: +1-713-366-8072 ext 3
UK Tel: +44-20-80991035 ext 3
Email: jamie@stimulussoft.com
Web: http://www.mailarchiva.com

To receive MailArchiva Enterprise Edition product announcements, send a message to: <mailarchiva-enterprise-edition-subscribe@stimulussoft.com>



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message