lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: IndexWriter.getReader.getVersion behavior
Date Fri, 26 Feb 2010 20:29:24 GMT
That should be fine!

Mike

On Fri, Feb 26, 2010 at 3:26 PM, Peter Keegan <peterlkeegan@gmail.com> wrote:
> Can  IW.waitForMerges be called between 'prepareCommit' and 'commit'? That's
> when the app calls 'getReader' to create external data.
>
> Peter
>
> On Fri, Feb 26, 2010 at 3:15 PM, Peter Keegan <peterlkeegan@gmail.com>wrote:
>
>> Great, I'll give it a try.
>> Thanks!
>>
>>
>> On Fri, Feb 26, 2010 at 3:11 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>> Note that it's a BG merge (not commit)...
>>>
>>> You can use the new (as of 2.9 I think) IndexWriter.waitForMerges API?
>>>  If you call that, then call .getReader().getVersion(), then close &
>>> open the writer, I think (but you better test to be sure!) the next
>>> .getReader().getVersion() should always match.
>>>
>>> Mike
>>>
>>> On Fri, Feb 26, 2010 at 2:40 PM, Peter Keegan <peterlkeegan@gmail.com>
>>> wrote:
>>> > Is there a way for the application to wait for the BG commit to finish
>>> > before it calls IW.close? If so, would this prevent the extra version?
>>> The
>>> > extra version causes the app. to think that the external data it
>>> committed
>>> > is out of synch with the index, which requires the app to do extra
>>> > processing to re-synch.
>>> >
>>> > Thanks,
>>> > Peter
>>> >
>>> >
>>> > On Fri, Feb 26, 2010 at 12:40 PM, Michael McCandless <
>>> > lucene@mikemccandless.com> wrote:
>>> >
>>> >> OK -- I can now see what happened.
>>> >>
>>> >> There was a merge still running, when you called IW.commit (Lucene
>>> >> Merge Thread #0).  Because IW.commit does not wait for BG merges to
>>> >> finish, but IW.close does (by default), this means you'll pick up an
>>> >> extra version whenever a merge is running when you call close.
>>> >>
>>> >> Mike
>>> >>
>>> >> On Thu, Feb 25, 2010 at 2:52 PM, Peter Keegan <peterlkeegan@gmail.com>
>>> >> wrote:
>>> >> > I'm pretty sure this output occurred when the version number skipped
>>> +1.
>>> >> > The line containing '============'. separates the close/open
>>> IndexWriter.
>>> >> >
>>> >> > IFD [Indexer]: setInfoStream
>>> >> >
>>> >>
>>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@646f9dd9
>>> >> > IW 9 [Indexer]: setInfoStream:
>>> >> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
>>> >> > pathname>\lresumes1.search.main.1 autoCommit=false
>>> >> >
>>> >>
>>> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@5be44512mergeScheduler
>>> >>
>>> =org.apache.lucene.index.ConcurrentMergeScheduler@6772cfdframBufferSizeMB
>>> >> =16.0
>>> >> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
>>> >> > maxFieldLength=100000 index=_a:C9780 _b:C1204->_b _c:C717->_b
>>> >> _d:C1220->_d
>>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>>> >> > IW 9 [Indexer]: flush at getReader
>>> >> > IW 9 [Indexer]:   flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 9 [Indexer]:   index before flush _a:C9780 _b:C1204->_b
>>> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > IW 9 [UpdWriterBuild : 9]: DW:   RAM: now flush @ usedMB=15.816
>>> >> > allocMB=15.816 deletesMB=0.203 triggerMB=16
>>> >> > IW 9 [UpdWriterBuild : 9]:   flush: segment=_j docStoreSegment=_j
>>> >> > docStoreOffset=0 flushDocs=true flushDeletes=false
>>> flushDocStores=false
>>> >> > numDocs=1456 numBufDelTerms=1456
>>> >> > IW 9 [UpdWriterBuild : 9]:   index before flush _a:C9780 _b:C1204->_b
>>> >> > _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f
>>> >> _h:C1291->_h
>>> >> > _i:C703->_h
>>> >> > IW 9 [UpdWriterBuild : 9]: DW: flush postings as segment _j
>>> numDocs=1456
>>> >> > IW 9 [UpdWriterBuild : 9]: DW:   oldRAMSize=16584704
>>> >> newFlushedSize=4969789
>>> >> > docs/MB=307.202 new/old=29.966%
>>> >> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments
;
>>> >> > isCommit = false]
>>> >> > IFD [UpdWriterBuild : 9]: now checkpoint "segments_b" [10 segments
;
>>> >> > isCommit = false]
>>> >> > IW 9 [UpdWriterBuild : 9]: LMP: findMerges: 10 segments
>>> >> > IW 9 [UpdWriterBuild : 9]: LMP:   level 6.863048 to 7.613048:
1
>>> segments
>>> >> > IW 9 [UpdWriterBuild : 9]: LMP:   level 6.2247195 to 6.696363:
9
>>> segments
>>> >> > IW 9 [UpdWriterBuild : 9]: CMS: now merge
>>> >> > IW 9 [UpdWriterBuild : 9]: CMS:   index: _a:C9780 _b:C1204->_b
>>> >> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j
>>> >> > IW 9 [UpdWriterBuild : 9]: CMS:   no more merges pending; now
return
>>> >> > IW 9 [Indexer]: prepareCommit: flush
>>> >> > IW 9 [Indexer]:   flush: segment=_k docStoreSegment=_j
>>> >> docStoreOffset=1456
>>> >> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=509
>>> >> > numBufDelTerms=509
>>> >> > IW 9 [Indexer]:   index before flush _a:C9780 _b:C1204->_b
>>> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j
>>> >> > IW 9 [Indexer]:   flush shared docStore segment _j
>>> >> > IW 9 [Indexer]: DW: closeDocStore: 2 files to flush to segment
_j
>>> >> > numDocs=1965
>>> >> > IW 9 [Indexer]: DW: flush postings as segment _k numDocs=509
>>> >> > IW 9 [Indexer]: DW:   oldRAMSize=7483392 newFlushedSize=1854970
>>> >> > docs/MB=287.727 new/old=24.788%
>>> >> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit
=
>>> >> false]
>>> >> > IW 9 [Indexer]: DW: apply 1965 buffered deleted terms and 0 deleted
>>> >> docIDs
>>> >> > and 0 deleted queries on 11 segments.
>>> >> > IFD [Indexer]: now checkpoint "segments_b" [11 segments ; isCommit
=
>>> >> false]
>>> >> > IW 9 [Indexer]: LMP: findMerges: 11 segments
>>> >> > IW 9 [Indexer]: LMP:   level 6.863048 to 7.613048: 1 segments
>>> >> > IW 9 [Indexer]: LMP:   level 6.2247195 to 6.696363: 10 segments
>>> >> > IW 9 [Indexer]: LMP:     1 to 11: add this merge
>>> >> > IW 9 [Indexer]: add merge to pendingMerges: _b:C1204->_b _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j [total 1 pending]
>>> >> > IW 9 [Indexer]: CMS: now merge
>>> >> > IW 9 [Indexer]: CMS:   index: _a:C9780 _b:C1204->_b _c:C717->_b
>>> >> _d:C1220->_d
>>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>>> >> _j:C1456->_j
>>> >> > _k:C509->_j
>>> >> > IW 9 [Indexer]: CMS:   consider merge _b:C1204->_b _c:C717->_b
>>> >> _d:C1220->_d
>>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>>> >> _j:C1456->_j
>>> >> > _k:C509->_j into _l [mergeDocStores]
>>> >> > IW 9 [Indexer]: CMS:     launch new thread [Lucene Merge Thread
#0]
>>> >> > IW 9 [Indexer]: CMS:   no more merges pending; now return
>>> >> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0
>>> >> > IW 9 [Indexer]: startCommit index=_a:C9780 _b:C1204->_b _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j changeCount=7
>>> >> > IW 9 [Lucene Merge Thread #0]: CMS:   merge thread: start
>>> >> > IW 9 [Indexer]: now sync _k.fnm
>>> >> > IW 9 [Lucene Merge Thread #0]: now merge
>>> >> >  merge=_b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d
_f:C1173->_f
>>> >> > _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j
into _l
>>> >> > [mergeDocStores]
>>> >> >  merge=org.apache.lucene.index.MergePolicy$OneMerge@5203ef34
>>> >> >  index=_a:C9780 _b:C1204->_b _c:C717->_b _d:C1220->_d
_e:C778->_d
>>> >> > _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h _j:C1456->_j
>>> >> _k:C509->_j
>>> >> > IW 9 [Lucene Merge Thread #0]: merging _b:C1204->_b _c:C717->_b
>>> >> _d:C1220->_d
>>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>>> >> _j:C1456->_j
>>> >> > _k:C509->_j into _l [mergeDocStores]
>>> >> > IW 9 [Indexer]: now sync _k.tii
>>> >> > IW 9 [Indexer]: now sync _k.tis
>>> >> > IW 9 [Lucene Merge Thread #0]: merge: total 9909 docs
>>> >> > IW 9 [Indexer]: now sync _j.tis
>>> >> > IW 9 [Indexer]: now sync _j.tii
>>> >> > IW 9 [Indexer]: now sync _j.fdx
>>> >> > IW 9 [Indexer]: now sync _j.fdt
>>> >> > IW 9 [Indexer]: now sync _k.prx
>>> >> > IW 9 [Indexer]: now sync _k.nrm
>>> >> > IW 9 [Indexer]: now sync _j.nrm
>>> >> > IW 9 [Indexer]: now sync _j.prx
>>> >> > IW 9 [Indexer]: now sync _j.fnm
>>> >> > IW 9 [Indexer]: now sync _k.frq
>>> >> > IW 9 [Indexer]: now sync _j.frq
>>> >> > IW 9 [Indexer]: done all syncs
>>> >> > IW 9 [Indexer]: flush at getReader
>>> >> > IW 9 [Indexer]:   flush: segment=null docStoreSegment=_k
>>> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 9 [Indexer]:   index before flush _a:C9780 _b:C1204->_b
>>> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j
>>> >> > IW 9 [Indexer]:   flush shared docStore segment _k
>>> >> > IW 9 [Indexer]: DW: closeDocStore: 0 files to flush to segment
_k
>>> >> numDocs=0
>>> >> > IW 9 [Indexer]: commit: start
>>> >> > IW 9 [Indexer]: commit: already prepared
>>> >> > IW 9 [Indexer]: commit: pendingCommit != null
>>> >> > IW 9 [Indexer]: commit: wrote segments file "segments_c"
>>> >> > IFD [Indexer]: now checkpoint "segments_c" [11 segments ; isCommit
=
>>> >> true]
>>> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_b"
>>> >> > IFD [Indexer]: delete "segments_b"
>>> >> > IW 9 [Indexer]: commit: done
>>> >> > IW 9 [Indexer]: flush at getReader
>>> >> > IW 9 [Indexer]:   flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 9 [Indexer]:   index before flush _a:C9780 _b:C1204->_b
>>> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j
>>> >> > IW 9 [Indexer]: now flush at close
>>> >> > IW 9 [Indexer]:   flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 9 [Indexer]:   index before flush _a:C9780 _b:C1204->_b
>>> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j
>>> >> > IW 9 [Indexer]: CMS: now merge
>>> >> > IW 9 [Indexer]: CMS:   index: _a:C9780 _b:C1204->_b _c:C717->_b
>>> >> _d:C1220->_d
>>> >> > _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h _i:C703->_h
>>> >> _j:C1456->_j
>>> >> > _k:C509->_j
>>> >> > IW 9 [Indexer]: CMS:   no more merges pending; now return
>>> >> > IW 9 [Lucene Merge Thread #0]: commitMerge: _b:C1204->_b _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores] index=_a:C9780
>>> >> > _b:C1204->_b _c:C717->_b _d:C1220->_d _e:C778->_d _f:C1173->_f
>>> >> _g:C858->_f
>>> >> > _h:C1291->_h _i:C703->_h _j:C1456->_j _k:C509->_j
>>> >> > IW 9 [Lucene Merge Thread #0]: commitMergeDeletes _b:C1204->_b
>>> >> _c:C717->_b
>>> >> > _d:C1220->_d _e:C778->_d _f:C1173->_f _g:C858->_f _h:C1291->_h
>>> >> _i:C703->_h
>>> >> > _j:C1456->_j _k:C509->_j into _l [mergeDocStores]
>>> >> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments
>>> ;
>>> >> > isCommit = false]
>>> >> > IFD [Lucene Merge Thread #0]: now checkpoint "segments_c" [2 segments
>>> ;
>>> >> > isCommit = false]
>>> >> > IW 9 [Lucene Merge Thread #0]: CMS:   merge thread: done
>>> >> > IW 9 [Indexer]: now call final commit()
>>> >> > IW 9 [Indexer]: startCommit(): start sizeInBytes=0
>>> >> > IW 9 [Indexer]: startCommit index=_a:C9780 _l:C9909 changeCount=8
>>> >> > IW 9 [Indexer]: now sync _l.nrm
>>> >> > IW 9 [Indexer]: now sync _l.fnm
>>> >> > IW 9 [Indexer]: now sync _l.prx
>>> >> > IW 9 [Indexer]: now sync _l.frq
>>> >> > IW 9 [Indexer]: now sync _l.tis
>>> >> > IW 9 [Indexer]: now sync _l.fdx
>>> >> > IW 9 [Indexer]: now sync _l.fdt
>>> >> > IW 9 [Indexer]: now sync _l.tii
>>> >> > IW 9 [Indexer]: done all syncs
>>> >> > IW 9 [Indexer]: commit: pendingCommit != null
>>> >> > IW 9 [Indexer]: commit: wrote segments file "segments_d"
>>> >> > IFD [Indexer]: now checkpoint "segments_d" [2 segments ; isCommit
=
>>> true]
>>> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_c"
>>> >> > IFD [Indexer]: delete "_h.prx"
>>> >> > IFD [Indexer]: delete "_f.tis"
>>> >> > IFD [Indexer]: delete "_g.tis"
>>> >> > IFD [Indexer]: delete "_d.tii"
>>> >> > IFD [Indexer]: delete "_e.frq"
>>> >> > IFD [Indexer]: delete "_k.fnm"
>>> >> > IFD [Indexer]: delete "_c.tii"
>>> >> > IFD [Indexer]: delete "_h.nrm"
>>> >> > IFD [Indexer]: delete "_i.tis"
>>> >> > IFD [Indexer]: delete "_k.tii"
>>> >> > IFD [Indexer]: delete "_h.fdt"
>>> >> > IFD [Indexer]: delete "_c.tis"
>>> >> > IFD [Indexer]: delete "_g.tii"
>>> >> > IFD [Indexer]: delete "_h.fdx"
>>> >> > IFD [Indexer]: delete "_c.nrm"
>>> >> > IFD [Indexer]: delete "_i.tii"
>>> >> > IFD [Indexer]: delete "_d.tis"
>>> >> > IFD [Indexer]: delete "_f.tii"
>>> >> > IFD [Indexer]: delete "_e.fnm"
>>> >> > IFD [Indexer]: delete "_k.tis"
>>> >> > IFD [Indexer]: delete "_d.frq"
>>> >> > IFD [Indexer]: delete "_f.nrm"
>>> >> > IFD [Indexer]: delete "_h.fnm"
>>> >> > IFD [Indexer]: delete "segments_c"
>>> >> > IFD [Indexer]: delete "_c.prx"
>>> >> > IFD [Indexer]: delete "_d.fnm"
>>> >> > IFD [Indexer]: delete "_h.frq"
>>> >> > IFD [Indexer]: delete "_h.tis"
>>> >> > IFD [Indexer]: delete "_f.fdt"
>>> >> > IFD [Indexer]: delete "_j.tis"
>>> >> > IFD [Indexer]: delete "_f.fdx"
>>> >> > IFD [Indexer]: delete "_g.nrm"
>>> >> > IFD [Indexer]: delete "_f.prx"
>>> >> > IFD [Indexer]: delete "_j.tii"
>>> >> > IFD [Indexer]: delete "_h.tii"
>>> >> > IFD [Indexer]: delete "_e.prx"
>>> >> > IFD [Indexer]: delete "_c.frq"
>>> >> > IFD [Indexer]: delete "_i.prx"
>>> >> > IFD [Indexer]: delete "_j.fdx"
>>> >> > IFD [Indexer]: delete "_i.nrm"
>>> >> > IFD [Indexer]: delete "_j.fdt"
>>> >> > IFD [Indexer]: delete "_k.prx"
>>> >> > IFD [Indexer]: delete "_b.nrm"
>>> >> > IFD [Indexer]: delete "_e.tis"
>>> >> > IFD [Indexer]: delete "_e.tii"
>>> >> > IFD [Indexer]: delete "_g.fnm"
>>> >> > IFD [Indexer]: delete "_k.nrm"
>>> >> > IFD [Indexer]: delete "_j.nrm"
>>> >> > IFD [Indexer]: delete "_j.prx"
>>> >> > IFD [Indexer]: delete "_e.nrm"
>>> >> > IFD [Indexer]: delete "_g.prx"
>>> >> > IFD [Indexer]: delete "_b.prx"
>>> >> > IFD [Indexer]: delete "_i.frq"
>>> >> > IFD [Indexer]: delete "_b.tis"
>>> >> > IFD [Indexer]: delete "_d.prx"
>>> >> > IFD [Indexer]: delete "_g.frq"
>>> >> > IFD [Indexer]: delete "_d.fdx"
>>> >> > IFD [Indexer]: delete "_b.tii"
>>> >> > IFD [Indexer]: delete "_b.frq"
>>> >> > IFD [Indexer]: delete "_d.fdt"
>>> >> > IFD [Indexer]: delete "_f.fnm"
>>> >> > IFD [Indexer]: delete "_j.fnm"
>>> >> > IFD [Indexer]: delete "_b.fnm"
>>> >> > IFD [Indexer]: delete "_d.nrm"
>>> >> > IFD [Indexer]: delete "_b.fdt"
>>> >> > IFD [Indexer]: delete "_k.frq"
>>> >> > IFD [Indexer]: delete "_c.fnm"
>>> >> > IFD [Indexer]: delete "_j.frq"
>>> >> > IFD [Indexer]: delete "_i.fnm"
>>> >> > IFD [Indexer]: delete "_f.frq"
>>> >> > IFD [Indexer]: delete "_b.fdx"
>>> >> > IW 9 [Indexer]: commit: done
>>> >> > IW 9 [Indexer]: at close: _a:C9780 _l:C9909
>>> >> >
>>> >> > =====================================================
>>> >> >
>>> >> > IFD [Indexer]: setInfoStream
>>> >> >
>>> >>
>>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@68d349c1
>>> >> > IW 10 [Indexer]: setInfoStream:
>>> >> > dir=org.apache.lucene.store.SimpleFSDirectory@<obfuscated
>>> >> > pathname>\lresumes1.search.main.1 autoCommit=false
>>> >> >
>>> >>
>>> mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@78f53abmergeScheduler
>>> >>
>>> =org.apache.lucene.index.ConcurrentMergeScheduler@52be036dramBufferSizeMB
>>> >> =16.0
>>> >> > maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
>>> >> > maxFieldLength=100000 index=_a:C9780 _l:C9909
>>> >> > IW 10 [Indexer]: flush at getReader
>>> >> > IW 10 [Indexer]:   flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 10 [Indexer]:   index before flush _a:C9780 _l:C9909
>>> >> > IW 10 [UpdWriterBuild : 10]: DW:   RAM: now flush @ usedMB=15.789
>>> >> > allocMB=15.789 deletesMB=0.212 triggerMB=16
>>> >> > IW 10 [UpdWriterBuild : 10]:   flush: segment=_m docStoreSegment=_m
>>> >> > docStoreOffset=0 flushDocs=true flushDeletes=false
>>> flushDocStores=false
>>> >> > numDocs=1526 numBufDelTerms=1526
>>> >> > IW 10 [UpdWriterBuild : 10]:   index before flush _a:C9780 _l:C9909
>>> >> > IW 10 [UpdWriterBuild : 10]: DW: flush postings as segment _m
>>> >> numDocs=1526
>>> >> > IW 10 [UpdWriterBuild : 10]: DW:   oldRAMSize=16556032
>>> >> > newFlushedSize=4788823 docs/MB=334.138 new/old=28.925%
>>> >> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments
;
>>> >> > isCommit = false]
>>> >> > IFD [UpdWriterBuild : 10]: now checkpoint "segments_d" [3 segments
;
>>> >> > isCommit = false]
>>> >> > IW 10 [UpdWriterBuild : 10]: LMP: findMerges: 3 segments
>>> >> > IW 10 [UpdWriterBuild : 10]: LMP:   level 6.8642592 to 7.6142592:
2
>>> >> segments
>>> >> > IW 10 [UpdWriterBuild : 10]: LMP:   level 6.2247195 to 6.6802545:
1
>>> >> segments
>>> >> > IW 10 [UpdWriterBuild : 10]: CMS: now merge
>>> >> > IW 10 [UpdWriterBuild : 10]: CMS:   index: _a:C9780 _l:C9909
>>> _m:C1526->_m
>>> >> > IW 10 [UpdWriterBuild : 10]: CMS:   no more merges pending; now
>>> return
>>> >> > IW 10 [Indexer]: prepareCommit: flush
>>> >> > IW 10 [Indexer]:   flush: segment=_n docStoreSegment=_m
>>> >> docStoreOffset=1526
>>> >> > flushDocs=true flushDeletes=true flushDocStores=true numDocs=501
>>> >> > numBufDelTerms=501
>>> >> > IW 10 [Indexer]:   index before flush _a:C9780 _l:C9909 _m:C1526->_m
>>> >> > IW 10 [Indexer]:   flush shared docStore segment _m
>>> >> > IW 10 [Indexer]: DW: closeDocStore: 2 files to flush to segment
_m
>>> >> > numDocs=2027
>>> >> > IW 10 [Indexer]: DW: flush postings as segment _n numDocs=501
>>> >> > IW 10 [Indexer]: DW:   oldRAMSize=6927360 newFlushedSize=1659342
>>> >> > docs/MB=316.593 new/old=23.953%
>>> >> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit
=
>>> >> false]
>>> >> > IW 10 [Indexer]: DW: apply 2027 buffered deleted terms and 0 deleted
>>> >> docIDs
>>> >> > and 0 deleted queries on 4 segments.
>>> >> > IFD [Indexer]: now checkpoint "segments_d" [4 segments ; isCommit
=
>>> >> false]
>>> >> > IW 10 [Indexer]: LMP: findMerges: 4 segments
>>> >> > IW 10 [Indexer]: LMP:   level 6.8642592 to 7.6142592: 2 segments
>>> >> > IW 10 [Indexer]: LMP:   level 6.2247195 to 6.6802545: 1 segments
>>> >> > IW 10 [Indexer]: LMP:   level -1.0 to 6.220011: 1 segments
>>> >> > IW 10 [Indexer]: CMS: now merge
>>> >> > IW 10 [Indexer]: CMS:   index: _a:C9780 _l:C9909 _m:C1526->_m
>>> _n:C501->_m
>>> >> > IW 10 [Indexer]: CMS:   no more merges pending; now return
>>> >> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0
>>> >> > IW 10 [Indexer]: startCommit index=_a:C9780 _l:C9909 _m:C1526->_m
>>> >> > _n:C501->_m changeCount=6
>>> >> > IW 10 [Indexer]: now sync _m.frq
>>> >> > IW 10 [Indexer]: now sync _m.prx
>>> >> > IW 10 [Indexer]: now sync _n.tis
>>> >> > IW 10 [Indexer]: now sync _m.tis
>>> >> > IW 10 [Indexer]: now sync _n.frq
>>> >> > IW 10 [Indexer]: now sync _n.tii
>>> >> > IW 10 [Indexer]: now sync _m.tii
>>> >> > IW 10 [Indexer]: now sync _m.fnm
>>> >> > IW 10 [Indexer]: now sync _n.fnm
>>> >> > IW 10 [Indexer]: now sync _n.nrm
>>> >> > IW 10 [Indexer]: now sync _n.prx
>>> >> > IW 10 [Indexer]: now sync _m.nrm
>>> >> > IW 10 [Indexer]: now sync _m.fdt
>>> >> > IW 10 [Indexer]: now sync _m.fdx
>>> >> > IW 10 [Indexer]: done all syncs
>>> >> > IW 10 [Indexer]: flush at getReader
>>> >> > IW 10 [Indexer]:   flush: segment=null docStoreSegment=_n
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=true numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 10 [Indexer]:   index before flush _a:C9780 _l:C9909 _m:C1526->_m
>>> >> > _n:C501->_m
>>> >> > IW 10 [Indexer]:   flush shared docStore segment _n
>>> >> > IW 10 [Indexer]: DW: closeDocStore: 0 files to flush to segment
_n
>>> >> numDocs=0
>>> >> > IW 10 [Indexer]: commit: start
>>> >> > IW 10 [Indexer]: commit: already prepared
>>> >> > IW 10 [Indexer]: commit: pendingCommit != null
>>> >> > IW 10 [Indexer]: commit: wrote segments file "segments_e"
>>> >> > IFD [Indexer]: now checkpoint "segments_e" [4 segments ; isCommit
=
>>> true]
>>> >> > IFD [Indexer]: deleteCommits: now decRef commit "segments_d"
>>> >> > IFD [Indexer]: delete "segments_d"
>>> >> > IW 10 [Indexer]: commit: done
>>> >> > IW 10 [Indexer]: flush at getReader
>>> >> > IW 10 [Indexer]:   flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 10 [Indexer]:   index before flush _a:C9780 _l:C9909 _m:C1526->_m
>>> >> > _n:C501->_m
>>> >> > IW 10 [Indexer]: now flush at close
>>> >> > IW 10 [Indexer]:   flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
>>> >> > numBufDelTerms=0
>>> >> > IW 10 [Indexer]:   index before flush _a:C9780 _l:C9909 _m:C1526->_m
>>> >> > _n:C501->_m
>>> >> > IW 10 [Indexer]: CMS: now merge
>>> >> > IW 10 [Indexer]: CMS:   index: _a:C9780 _l:C9909 _m:C1526->_m
>>> _n:C501->_m
>>> >> > IW 10 [Indexer]: CMS:   no more merges pending; now return
>>> >> > IW 10 [Indexer]: now call final commit()
>>> >> > IW 10 [Indexer]: startCommit(): start sizeInBytes=0
>>> >> > IW 10 [Indexer]:   skip startCommit(): no changes pending
>>> >> > IW 10 [Indexer]: commit: pendingCommit == null; skip
>>> >> > IW 10 [Indexer]: commit: done
>>> >> > IW 10 [Indexer]: at close: _a:C9780 _l:C9909 _m:C1526->_m _n:C501->_m
>>> >> >
>>> >> >
>>> >> > Peter
>>> >> >
>>> >> > On Thu, Feb 25, 2010 at 1:13 PM, Michael McCandless <
>>> >> > lucene@mikemccandless.com> wrote:
>>> >> >
>>> >> >> Do you know the place in the infoStream output where you got
a
>>> reader
>>> >> >> with the wrong (unexplained extra +1) version?  If so, can
you post
>>> >> >> the infoStream output up to that point?
>>> >> >>
>>> >> >> Mike
>>> >> >>
>>> >> >> On Thu, Feb 25, 2010 at 10:22 AM, Peter Keegan <
>>> peterlkeegan@gmail.com>
>>> >> >> wrote:
>>> >> >> > I've reproduced this and I have a bunch of infoStream
log files.
>>> Since
>>> >> >> the
>>> >> >> > messages have no timestamps, it's hard to tell where the
relevant
>>> >> entries
>>> >> >> > are. What should I be looking for?
>>> >> >> >
>>> >> >> > Peter
>>> >> >> >
>>> >> >> > On Mon, Feb 22, 2010 at 3:58 PM, Peter Keegan <
>>> peterlkeegan@gmail.com
>>> >> >> >wrote:
>>> >> >> >
>>> >> >> >> I'm pretty sure there are flushes and segment merges
going on,
>>> but as
>>> >> >> you
>>> >> >> >> said, that shouldn't affect the version increment.
I'll see what
>>> I
>>> >> can
>>> >> >> do to
>>> >> >> >> get infoStream output.
>>> >> >> >>
>>> >> >> >> Thanks,
>>> >> >> >> Peter
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> On Mon, Feb 22, 2010 at 2:30 PM, Michael McCandless
<
>>> >> >> >> lucene@mikemccandless.com> wrote:
>>> >> >> >>
>>> >> >> >>> Well I'm at a loss then.  The version should
only increment on
>>> >> commit.
>>> >> >> >>>
>>> >> >> >>> Can you make it all happen when infoStream is
on, and post back?
>>> >> >> >>>
>>> >> >> >>> Mike
>>> >> >> >>>
>>> >> >> >>> On Mon, Feb 22, 2010 at 12:35 PM, Peter Keegan
<
>>> >> peterlkeegan@gmail.com
>>> >> >> >
>>> >> >> >>> wrote:
>>> >> >> >>> > Only one writer thread and one writer process.
>>> >> >> >>> > I'm calling IndexWriter(Directory d, Analyzer
a, boolean
>>> create,
>>> >> >> >>> > MaxFieldLength mfl), which sets autocommit=false.
>>> >> >> >>> >
>>> >> >> >>> > Peter
>>> >> >> >>> >
>>> >> >> >>> > On Mon, Feb 22, 2010 at 12:24 PM, Michael
McCandless <
>>> >> >> >>> > lucene@mikemccandless.com> wrote:
>>> >> >> >>> >
>>> >> >> >>> >> That's curious.
>>> >> >> >>> >>
>>> >> >> >>> >> It's only on prepareCommit (or, commit,
if you didn't first
>>> >> prepare,
>>> >> >> >>> >> since that will call prepareCommit internally)
that this
>>> version
>>> >> >> >>> >> should increase.
>>> >> >> >>> >>
>>> >> >> >>> >> Is there only 1 thread doing this?
>>> >> >> >>> >>
>>> >> >> >>> >> Oh, and, are you passing false for autoCommit?
>>> >> >> >>> >>
>>> >> >> >>> >> Mike
>>> >> >> >>> >>
>>> >> >> >>> >> On Mon, Feb 22, 2010 at 11:43 AM, Peter
Keegan <
>>> >> >> peterlkeegan@gmail.com
>>> >> >> >>> >
>>> >> >> >>> >> wrote:
>>> >> >> >>> >> > Using Lucene 2.9.1, I have the following
pseudocode which
>>> gets
>>> >> >> >>> repeated
>>> >> >> >>> >> at
>>> >> >> >>> >> > regular intervals:
>>> >> >> >>> >> >
>>> >> >> >>> >> > 1. FSDirectory dir = FSDirectory.open(java.io.File);
>>> >> >> >>> >> > 2. dir.setLockFactory(new SingleInstanceLockFactory());
>>> >> >> >>> >> > 3. IndexWriter writer = new IndexWriter(dir,
Analyzer,
>>> false,
>>> >> >> >>> >> maxFieldLen)
>>> >> >> >>> >> > 4. writer.getReader().getVersion();
>>> >> >> >>> >> > 5. writer.prepareCommit();
>>> >> >> >>> >> > 6. writer.getReader().getVersion();
>>> >> >> >>> >> > 7. writer.commit();
>>> >> >> >>> >> > 8. writer.close();
>>> >> >> >>> >> >
>>> >> >> >>> >> > I'm using the version number to
keep external data in synch
>>> >> with
>>> >> >> the
>>> >> >> >>> >> index.
>>> >> >> >>> >> > Usually, the version number from
(6) is 1 greater than from
>>> (4)
>>> >> >> and
>>> >> >> >>> the
>>> >> >> >>> >> > version from (4) equals the version
from the previous (6).
>>> At
>>> >> >> least
>>> >> >> >>> once
>>> >> >> >>> >> a
>>> >> >> >>> >> > day, however, the version from (4)
is 1 greater than from
>>> the
>>> >> >> >>> previous
>>> >> >> >>> >> (6).
>>> >> >> >>> >> > What would explain this sporadic
behavior of version
>>> numbers?
>>> >> >> >>> >> >
>>> >> >> >>> >> > Thanks,
>>> >> >> >>> >> > Peter
>>> >> >> >>> >> >
>>> >> >> >>> >>
>>> >> >> >>> >>
>>> >> >>
>>> ---------------------------------------------------------------------
>>> >> >> >>> >> To unsubscribe, e-mail:
>>> java-user-unsubscribe@lucene.apache.org
>>> >> >> >>> >> For additional commands, e-mail:
>>> >> java-user-help@lucene.apache.org
>>> >> >> >>> >>
>>> >> >> >>> >>
>>> >> >> >>> >
>>> >> >> >>>
>>> >> >> >>>
>>> >> ---------------------------------------------------------------------
>>> >> >> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> >> >> >>> For additional commands, e-mail:
>>> java-user-help@lucene.apache.org
>>> >> >> >>>
>>> >> >> >>>
>>> >> >> >>
>>> >> >> >
>>> >> >>
>>> >> >>
>>> ---------------------------------------------------------------------
>>> >> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> >> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >> >>
>>> >> >>
>>> >> >
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >>
>>> >>
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message