lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless" <luc...@mikemccandless.com>
Subject Re: Lucene 2.3.1 IndexReader and IndexWriter Synchronization Issues
Date Sat, 29 Mar 2008 10:43:56 GMT
OK indeed from your code you are manually removing files from the
index (the "deleteIndex" method).  You shouldn't do that.

I think what's happening is deleteIndex is called when an IndexReader
has the index open (do you see those "failed to delete file in
existing index" log messages?), then you open a writer on that
directory.  That would exactly explain your exception.

The fix is simple: never remove files from the index yourself.  Let
IndexWriter do it, by opening with create=true.  It will take care of
removing the old files, retrying over time, etc.

And, most importantly, this allows you to create a new index in a
directory even while an IndexReader is still using it.  The reader
will be fine, and then when it reopens it will switch to the new
index.

Lucene 2.3 can read 2.2 indices but not vice versa, so you can't
downgrade unless you completely remove your index.

Mike

Jamie <jamie@stimulussoft.com> wrote:
> By the way, the line numbers in the last email apply to the SVN trunk
>  code. I switch over to the latest code to see if this would make a
>  difference. Alas, the problem still exists!
>
>
>
>  Jamie wrote:
>  > Hi Michael
>  >
>  > Glad you asked and thanks in advance for your help! I am trying to
>  > reindex 325 emails. Its a pretty straight forward use of Lucene. I
>  > create an IndexWriter, write a whole bunch of documents and then close
>  > the Index every 2 seconds. See the attached source code. In case, you
>  > are wondering there is only ever one VolumeIndex ever created.
>  > Following, there is only ever one IndexWriter created in the entire
>  > application. After further testing I dont think the problem is related
>  > to IndexReader since the problem still occurs even if I dont execute a
>  > search. The amount emails indexed varies between around 305 and 325.
>  > The exception that gets thrown prevents the emails from being indexed.
>  > I've tried  removing the synchronization and doing a simple:
>  >
>  > synchronized(this) {
>  >    openIndex()
>  >    write()
>  >    closeIndex()
>  > }
>  >
>  > i.e. open and close the index between every document write and the
>  > problem still appears.
>  >
>  > For a recap, here is the exception:
>  >
>  > java.io.IOException: Cannot overwrite: C:\index9121\_1.cfs
>  >    at
>  > org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:433)
>  >    at
>  > org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:150)
>  >
>  >    at
>  > org.apache.lucene.index.DocumentsWriter.createCompoundFile(DocumentsWriter.java:587)
>  >
>  >    at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3251)
>  >    at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3110)
>  >    at
>  > org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1659)
>  >    at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1633)
>  >    at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1601)
>  >
>  > I cleared out the logs and executed another reindex and the following
>  > debug information was outputted by Lucene. Unfortunately, the debug
>  > output does not include any information on the error. I am not sure
>  > why this is the case, but I can assure you that the log information
>  > was generated by the reindexing operation. Any ideas on what this
>  > might be? Can you suggest a workaround?
>  >
>  > As an aside note, is Lucene 2.2.0 compatible with Lucene 2.3.0
>  > indexes? If I cant sort this out in the next couple of days, I may
>  > need to switch everyone back to Lucene 2.2.0 temporarily until this
>  > problem is resolved.
>  >
>  > IFD [http-8090-2]: setInfoStream
>  > deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@1ebf305
>  >
>  > IW 22 [http-8090-2]: setInfoStream:
>  > dir=org.apache.lucene.store.FSDirectory@C:\index9121 autoCommit=false
>  > mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@19ffd6f
>  > mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@b799cf
>  > ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
>  > maxFieldLength=50000 index=_0:C54 _1:C82 _2:C14 _3:c36 _4:c6 _5:c52
>  > _6:c53
>  > IW 22 [Timer-0]: now flush at close
>  > IW 22 [Timer-0]:   flush: segment=_7 docStoreSegment=_7
>  > docStoreOffset=0 flushDocs=true flushDeletes=true flushDocStores=true
>  > numDocs=28 numBufDelTerms=0
>  > IW 22 [Timer-0]:   index before flush _0:C54 _1:C82 _2:C14 _3:c36
>  > _4:c6 _5:c52 _6:c53
>  > IW 22 [Timer-0]: DW: flush postings as segment _7 numDocs=28
>  > IW 22 [Timer-0]: DW: closeDocStore: 2 files to flush to segment _7
>  > numDocs=28
>  > IW 22 [Timer-0]: DW:   oldRAMSize=245760 newFlushedSize=38119
>  > docs/MB=770.223 new/old=15.511%
>  > IFD [Timer-0]: now checkpoint "segments_8" [8 segments ; isCommit =
>  > false]
>  > IFD [Timer-0]: delete pending file _f.cfs
>  > IFD [Timer-0]: delete "_f.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _h.cfs
>  > IFD [Timer-0]: delete "_h.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _d.cfs
>  > IFD [Timer-0]: delete "_d.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _c.cfs
>  > IFD [Timer-0]: delete "_c.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _b.cfs
>  > IFD [Timer-0]: delete "_b.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _g.cfs
>  > IFD [Timer-0]: delete "_g.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _e.cfs
>  > IFD [Timer-0]: delete "_e.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: now checkpoint "segments_8" [8 segments ; isCommit =
>  > false]
>  > IFD [Timer-0]: delete pending file _f.cfs
>  > IFD [Timer-0]: delete "_f.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _h.cfs
>  > IFD [Timer-0]: delete "_h.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _d.cfs
>  > IFD [Timer-0]: delete "_d.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _c.cfs
>  > IFD [Timer-0]: delete "_c.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _b.cfs
>  > IFD [Timer-0]: delete "_b.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _g.cfs
>  > IFD [Timer-0]: delete "_g.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _e.cfs
>  > IFD [Timer-0]: delete "_e.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete "_7.fnm"
>  > IFD [Timer-0]: delete "_7.frq"
>  > IFD [Timer-0]: delete "_7.prx"
>  > IFD [Timer-0]: delete "_7.tis"
>  > IFD [Timer-0]: delete "_7.tii"
>  > IFD [Timer-0]: delete "_7.nrm"
>  > IFD [Timer-0]: delete "_7.fdx"
>  > IFD [Timer-0]: delete "_7.fdt"
>  > IW 22 [Timer-0]: LMP: findMerges: 8 segments
>  > IW 22 [Timer-0]: LMP:   level -1.0 to 5.202747: 8 segments
>  > IW 22 [Timer-0]: CMS: now merge
>  > IW 22 [Timer-0]: CMS:   index: _0:C54 _1:C82 _2:C14 _3:c36 _4:c6
>  > _5:c52 _6:c53 _7:c28
>  > IW 22 [Timer-0]: CMS:   no more merges pending; now return
>  > IW 22 [Timer-0]: now call final commit()
>  > IW 22 [Timer-0]: start commit() skipWait=true sizeInBytes=0
>  > IW 22 [Timer-0]: commit index=_0:C54 _1:C82 _2:C14 _3:c36 _4:c6 _5:c52
>  > _6:c53 _7:c28
>  > IW 22 [Timer-0]: now sync _7.cfs
>  > IW 22 [Timer-0]: commit complete
>  > IFD [Timer-0]: now checkpoint "segments_9" [8 segments ; isCommit = true]
>  > IFD [Timer-0]: delete pending file _f.cfs
>  > IFD [Timer-0]: delete "_f.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _h.cfs
>  > IFD [Timer-0]: delete "_h.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _d.cfs
>  > IFD [Timer-0]: delete "_d.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _c.cfs
>  > IFD [Timer-0]: delete "_c.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _b.cfs
>  > IFD [Timer-0]: delete "_b.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _g.cfs
>  > IFD [Timer-0]: delete "_g.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _e.cfs
>  > IFD [Timer-0]: delete "_e.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: deleteCommits: now decRef commit "segments_8"
>  > IFD [Timer-0]: delete "segments_8"
>  > IW 22 [Timer-0]: done all syncs
>  > IW 22 [Timer-0]: at close: _0:C54 _1:C82 _2:C14 _3:c36 _4:c6 _5:c52
>  > _6:c53 _7:c28
>  > IFD [Timer-0]: delete pending file _f.cfs
>  > IFD [Timer-0]: delete "_f.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_f.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_f.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _h.cfs
>  > IFD [Timer-0]: delete "_h.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_h.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_h.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _d.cfs
>  > IFD [Timer-0]: delete "_d.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_d.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_d.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _c.cfs
>  > IFD [Timer-0]: delete "_c.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_c.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_c.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _b.cfs
>  > IFD [Timer-0]: delete "_b.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_b.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_b.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _g.cfs
>  > IFD [Timer-0]: delete "_g.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_g.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_g.cfs; Will re-try
>  > later.
>  > IFD [Timer-0]: delete pending file _e.cfs
>  > IFD [Timer-0]: delete "_e.cfs"
>  > IFD [Timer-0]: IndexFileDeleter: unable to remove file "_e.cfs":
>  > java.io.IOException: Cannot delete C:\index9121\_e.cfs; Will re-try
>  > later.
>  >
>  >
>  > Michael McCandless wrote:
>  >>
>  >> That use case of Lucene should be fine, ie no further synchronization
>  >> should be necessary.
>  >>
>  >> Your debug output below is great, but it doesn't seem to cover the
>  >> occurrence of that exception.  Can you post the full debug output?
>  >>
>  >> Can you describe how you reindex in more detail?  Are you manually
>  >> removing files from the index and then opening a new IndexWriter with
>  >> create=true?
>  >>
>  >> Mike
>  >
>  > ------------------------------------------------------------------------
>
> >
>  > ---------------------------------------------------------------------
>  > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>  > For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>  --
>  Stimulus Software - MailArchiva
>  Email Archiving And Compliance
>  USA Tel: +1-713-366-8072 ext 3
>  UK Tel: +44-20-80991035 ext 3
>  Email: jamie@stimulussoft.com
>  Web: http://www.mailarchiva.com
>
>  To receive MailArchiva Enterprise Edition product announcements, send a message to:
<mailarchiva-enterprise-edition-subscribe@stimulussoft.com>
>
>
>
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>  For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message