lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: Index Corruption with Lucene 2.9.3
Date Mon, 14 Nov 2011 22:38:31 GMT
One addition:

Maybe you should update your antique Java version from the year 2007
(1.6.0_02) to something more up-to-date and maybe use 64 bit with mmap on a
local filesystem for such a large index.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, November 14, 2011 11:33 PM
> To: java-user@lucene.apache.org
> Subject: RE: Index Corruption with Lucene 2.9.3
> 
> Hi,
> 
> In general it's a bad idea to use Lucene on network-mounted drives. E.g.,
NFS is
> heavily broken with the file locking used by Lucene (NIO does not work at
all,
> and file-based lock support fails because directory updated may not be
visible
> at all times, or are visible before files are flushed - happens-before is
violated).
> 
> This can lead to index corruption; you should use local disks, especially
as they
> are much faster.
> 
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
> 
> > -----Original Message-----
> > From: nishesh.gupta@emc.com [mailto:nishesh.gupta@emc.com]
> > Sent: Monday, November 14, 2011 10:37 PM
> > To: java-user@lucene.apache.org
> > Subject: Index Corruption with Lucene 2.9.3
> >
> > We are seeing Index corruption very often with version 2.9.3. Our
> > indexing process is on Linux ( centos 5 ). Index is created on a
> > mounted drive
> which is a
> > shared drive from Windows 2008 server running in a VM. We generally
> > see index corruption in merge or optimize after indexing runs
> > continuously for
> 6-7
> > hrs with index size reaching around 7-8GB. To reproduce the
> > corruptions sooner, I have placed the merge ( maybemerge ) call
> > immediately after addIndex is called. We have a final index which is
> > in the mounted drive,
> we
> > always add documents to an local intermediate index and then call add
> index
> > and merge to the final index.
> >
> > The exception that I get -
> >
> > 2011-11-11 15:19:16,929 [MC:10.10.176.148-1321045422606-204
> > FS:emag_393219_0] ERROR indexer  - MergeWithFiler: MC: 393219, shard
> > 0, guid 10.10.176.148-1321045422606-204: Error in
> > addIndex()/kazMaybeMerge():
> > /sideline/fs_393219/cas/search/index_0/primary, java.io.IOException:
> > background merge hit exception: _1t:c262436
> > _10.10.176.148-1321045422606-
> > 204_0:cx4000 into _1u [optimize] [mergeDocStores] % STACK:
> >      org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2932)
> >      org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2867)
> >      org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2837)
> >
> > org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:3652)
> >
> > com.kazeon.search.indexingengine.context.MergerContext.mergeWithFilerI
> > nde
> > x(MergerContext.java:1004)
> >
> > com.kazeon.search.indexingengine.context.MergerContext.mergeWithFilerI
> > nde
> > x(MergerContext.java:1094)
> >
> > com.kazeon.search.indexingengine.context.MergerContext.mergeWithFiler(
> > Me
> > rgerContext.java:1140)
> >
> > com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIn
> > dex
> >
> OptimizeAndCompressModifier.modifyStateAux(LocalIndexOptimizeAndCompr
> > essModifier.java:375)
> >
> > com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIn
> > dex
> >
> OptimizeAndCompressModifier.mergeAllICs(LocalIndexOptimizeAndCompress
> > Modifier.java:181)
> >
> > com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIn
> > dex
> >
> OptimizeAndCompressModifier.modifyState(LocalIndexOptimizeAndCompress
> > Modifier.java:106)
> >      com.kazeon.util.scoreboard.WorkerThread.run(WorkerThread.java:31)
> >
> >
> > CheckIndex command shows the following output -
> >
> > NOTE: testing will be more thorough if you run java with '-
> > ea:org.apache.lucene...', so assertions are enabled
> >
> > Opening index @ .
> >
> > Segments file=segments_23 numSegments=1
> version=FORMAT_DIAGNOSTICS
> > [Lucene 2.9]
> >   1 of 1: name=_1t docCount=262436
> >     compound=true
> >     hasProx=true
> >     numFiles=1
> >     size (MB)=937.835
> >     diagnostics = {optimize=true, mergeFactor=2,
> os.version=2.6.18-92.1.18.el5,
> > os=Linux, mergeDocStores=true, lucene.version=2.9.3-dev, source=merge,
> > os.arch=i386, java.version=1.6.0_02, java.vendor=Sun Microsystems Inc.}
> >     no deletions
> >     test: open reader.........OK
> >     test: fields..............OK [79 fields]
> >     test: field norms.........OK [79 fields]
> >     test: terms, freq, prox...ERROR [term fulltext:creativecommons:
> > doc
> 262603
> > >= maxDoc 262436]
> > java.lang.RuntimeException: term fulltext:creativecommons: doc 262603
> > >= maxDoc 262436
> >     at
> org.apache.lucene.index.CheckIndex.testTermIndex(CheckIndex.java:646)
> >     at
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:530)
> >     at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)
> >     test: stored fields.......OK [524872 total field count; avg 2
> > fields
> per doc]
> >     test: term vectors........OK [0 total vector count; avg 0
> > term/freq
> vector fields
> > per doc] FAILED
> >     WARNING: fixIndex() would remove reference to this segment; full
> > exception:
> > java.lang.RuntimeException: Term Index test failed
> >     at
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:543)
> >     at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)
> >
> > WARNING: 1 broken segments (containing 262436 documents) detected
> > WARNING: would write new segments file, and 262436 documents would be
> > lost, if -fix were specified
> >
> >
> >
> > Lucene Indexwriter infostream around the corrupt segment _1t shows -
> >
> > 9620 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > setInfoStream: dir=org.apache.lucene.store.S
> > impleFSDirectory@/cas/fs_589826/sideline/fs_393219/search/index_0/prim
> > ary
> > autoCommit=true mergePolicy=org.
> > apache.lucene.index.LogByteSizeMergePolicy@2ac02f<mailto:apache.lucene
> > .in
> > dex.LogByteSizeMergePolicy@2ac02f>
> > mergeScheduler=org.apache.lucene.index.ConcurrentMergeSc
> > heduler@af7be8 ramBufferSizeMB=16.0 maxBufferedDocs=-1
> > maxBuffereDeleteTerms=-1 maxFieldLength=10000 index      =_1s:c261077
> > 9621 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > setMaxBufferedDocs 10000
> > 9622 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush at addIndexes
> > 9623 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: now pause all indexing threads
> > 9624 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: segment=null docStoreSegment=null d      ocStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > 9625 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > index before flush _1s:c261077
> > 9626 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> now
> > start transaction
> > 9627 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush at startTransaction
> > 9628 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: now pause all indexing threads
> > 9629 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: segment=null docStoreSegment=null d      ocStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > 9630 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > index before flush _1s:c261077
> > 9631 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > optimize: index now _1s:c261077 _10.10.176.1      48-1321053007682-
> > 361_0:cx1359**
> > 9632 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: now pause all indexing threads
> > 9633 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: segment=null docStoreSegment=null d      ocStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > 9634 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > index before flush _1s:c261077 _10.10.176.      148-1321053007682-
> > 361_0:cx1359**
> > 9635 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> add
> > merge to pendingMerges: _1s:c261077 _10.      10.176.148-1321053007682-
> > 361_0:cx1359 [optimize] [total 1 pending]
> > 9636 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS: now merge
> > 9637 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   index: _1s:c261077 _10.10.176.148-132
> 1053007682-361_0:cx1359**
> > 9638 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   consider merge _1s:c261077 _10.10.176      .148-1321053007682-
> > 361_0:cx1359 into _1t [optimize] [mergeDocStores]
> > 9639 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:     launch new thread [Lucene Merge Thr      ead #0]
> > 9640 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   no more merges pending; now return
> > 9641 IW 130 [Lucene Merge Thread #0]: CMS:   merge thread: start
> > 9642 IW 130 [Lucene Merge Thread #0]: now merge
> > 9643   merge=_1s:c261077 _10.10.176.148-1321053007682-361_0:cx1359
> into
> > _1t [optimize] [mergeDocStores]
> > 9644
> >
> merge=org.apache.lucene.index.MergePolicy$OneMerge@116eedb<mailto:m
> > erge=org.apache.lucene.index.MergePolicy$OneMerge@116eedb>
> > 9645   index=_1s:c261077 _10.10.176.148-1321053007682-361_0:cx1359**
> > 9646 IW 130 [Lucene Merge Thread #0]: merging _1s:c261077
> _10.10.176.148-
> > 1321053007682-361_0:cx1359 into _1t [o      ptimize] [mergeDocStores]
> > 9647 IW 130 [Lucene Merge Thread #0]: merge: total 262436 docs
> > 9648 IW 130 [Lucene Merge Thread #0]: commitMerge: _1s:c261077
> > _10.10.176.148-1321053007682-361_0:cx1359 into _      1t [optimize]
> > [mergeDocStores] index=_1s:c261077 _10.10.176.148-1321053007682-
> > 361_0:cx1359**
> > 9649 IW 130 [Lucene Merge Thread #0]: commitMergeDeletes _1s:c261077
> > _10.10.176.148-1321053007682-361_0:cx1359       into _1t [optimize]
> > [mergeDocStores]
> > 9650 IFD [Lucene Merge Thread #0]: now checkpoint "segments_1u" [1
> > segments ; isCommit = false]
> > 9651 IFD [Lucene Merge Thread #0]: now checkpoint "segments_1u" [1
> > segments ; isCommit = false]
> > 9652 IFD [Lucene Merge Thread #0]: now checkpoint "segments_1u" [1
> > segments ; isCommit = false]
> > 9653 IFD [Lucene Merge Thread #0]: delete "_1t.fnm"
> > 9654 IFD [Lucene Merge Thread #0]: delete "_1t.frq"
> > 9655 IFD [Lucene Merge Thread #0]: delete "_1t.prx"
> > 9656 IFD [Lucene Merge Thread #0]: delete "_1t.tis"
> > 9657 IFD [Lucene Merge Thread #0]: delete "_1t.tii"
> > 9658 IFD [Lucene Merge Thread #0]: delete "_1t.nrm"
> > 9659 IFD [Lucene Merge Thread #0]: delete "_1t.fdx"
> > 9660 IFD [Lucene Merge Thread #0]: delete "_1t.fdt"
> > 9661 IW 130 [Lucene Merge Thread #0]: CMS:   merge thread: done
> > 9662 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> now
> > commit transaction
> > 9663 IFD [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]: now
> > checkpoint "segments_1u" [1 segments ; isCo      mmit = false]
> > 9664 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > startCommit(): start sizeInBytes=0
> > 9665 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > startCommit index=_1t:c262436 changeCount=4
> > 9666 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> now
> > sync _1t.cfs
> > 9667 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > done all syncs
> > 9668 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > commit: pendingCommit != null
> > 9669 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > commit: wrote segments file "segments_1v"
> > 9670 IFD [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]: now
> > checkpoint "segments_1v" [1 segments ; isCo      mmit = true]
> > 9671 IFD [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > deleteCommits: now decRef commit "segments_1u"
> > 9672 IFD [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> delete
> > "_1s.cfs"
> > 9673 IFD [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> delete
> > "segments_1u"
> > 9674 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > commit: done
> > 9675 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > LMP: findMerges: 1 segments
> > 9676 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > LMP:   level 8.242726 to 8.992726: 1 segment      s
> > 9677 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS: now merge
> > 9678 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   index: _1t:c262436
> > 9679 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   no more merges pending; now return
> > 9680 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> now
> > flush at close
> > 9681 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: now pause all indexing threads
> > 9682 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: segment=null docStoreSegment=null d      ocStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > 9683 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > index before flush _1t:c262436
> > 9684 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS: now merge
> > 9685 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   index: _1t:c262436
> > 9686 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   no more merges pending; now return
> > 9687 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> now
> > call final commit()
> > 9688 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > startCommit(): start sizeInBytes=0
> > 9689 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > skip startCommit(): no changes pending
> > 9690 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > commit: pendingCommit == null; skip
> > 9691 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > commit: done
> > 9692 IW 130 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]: at
> > close: _1t:c262436
> > 9693 IFD [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > setInfoStream deletionPolicy=org.apache.lucene.
> >
> index.KeepOnlyLastCommitDeletionPolicy@28cdb8<mailto:index.KeepOnlyLas
> > t
> > CommitDeletionPolicy@28cdb8>
> > 9694 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > setInfoStream: dir=org.apache.lucene.store.S
> > impleFSDirectory@/cas/fs_589826/sideline/fs_393219/search/index_0/seco
> > nd
> > ary autoCommit=true mergePolicy=or
> > g.apache.lucene.index.LogByteSizeMergePolicy@da05c8<mailto:g.apache.lu
> > ce ne.index.LogByteSizeMergePolicy@da05c8>
> > mergeScheduler=org.apache.lucene.index.ConcurrentMerge
> > Scheduler@1b1c02b ramBufferSizeMB=16.0 maxBufferedDocs=-1
> > maxBuffereDeleteTerms=-1 maxFieldLength=10000 in      dex=_1s:c261077
> > 9695 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > setMaxBufferedDocs 10000
> > 9696 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush at addIndexes
> > 9697 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: now pause all indexing threads
> > 9698 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: segment=null docStoreSegment=null d      ocStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > 9699 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > index before flush _1s:c261077
> > 9700 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> now
> > start transaction
> > 9701 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush at startTransaction
> > 9702 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: now pause all indexing threads
> > 9703 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: segment=null docStoreSegment=null d      ocStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > 9704 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > index before flush _1s:c261077
> > 9705 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > optimize: index now _1s:c261077 _10.10.176.1      48-1321053007682-
> > 361_0:cx1359**
> > 9706 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: now pause all indexing threads
> > 9707 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: segment=null docStoreSegment=null d      ocStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > 9708 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > index before flush _1s:c261077 _10.10.176.      148-1321053007682-
> > 361_0:cx1359**
> > 9709 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> add
> > merge to pendingMerges: _1s:c261077 _10.      10.176.148-1321053007682-
> > 361_0:cx1359 [optimize] [total 1 pending]
> > 9710 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS: now merge
> > 9711 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   index: _1s:c261077 _10.10.176.148-132
> 1053007682-361_0:cx1359**
> > 9712 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   consider merge _1s:c261077 _10.10.176      .148-1321053007682-
> > 361_0:cx1359 into _1t [optimize] [mergeDocStores]
> > 9713 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:     launch new thread [Lucene Merge Thr      ead #0]
> > 9714 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   no more merges pending; now return
> > 9715 IW 131 [Lucene Merge Thread #0]: CMS:   merge thread: start
> > 9716 IW 131 [Lucene Merge Thread #0]: now merge
> > 9717   merge=_1s:c261077 _10.10.176.148-1321053007682-361_0:cx1359
> into
> > _1t [optimize] [mergeDocStores]
> > 9718
> >
> merge=org.apache.lucene.index.MergePolicy$OneMerge@15aa40f<mailto:me
> > rge=org.apache.lucene.index.MergePolicy$OneMerge@15aa40f>
> > 9719   index=_1s:c261077 _10.10.176.148-1321053007682-361_0:cx1359**
> > 9720 IW 131 [Lucene Merge Thread #0]: merging _1s:c261077
> _10.10.176.148-
> > 1321053007682-361_0:cx1359 into _1t [o      ptimize] [mergeDocStores]
> > 9721 IW 131 [Lucene Merge Thread #0]: merge: total 262436 docs
> > 9722 IW 131 [Lucene Merge Thread #0]: commitMerge: _1s:c261077
> > _10.10.176.148-1321053007682-361_0:cx1359 into _      1t [optimize]
> > [mergeDocStores] index=_1s:c261077 _10.10.176.148-1321053007682-
> > 361_0:cx1359**
> > 9723 IW 131 [Lucene Merge Thread #0]: commitMergeDeletes _1s:c261077
> > _10.10.176.148-1321053007682-361_0:cx1359       into _1t [optimize]
> > [mergeDocStores]
> > 9724 IFD [Lucene Merge Thread #0]: now checkpoint "segments_1u" [1
> > segments ; isCommit = false]
> > 9725 IFD [Lucene Merge Thread #0]: now checkpoint "segments_1u" [1
> > segments ; isCommit = false]
> > 9726 IFD [Lucene Merge Thread #0]: now checkpoint "segments_1u" [1
> > segments ; isCommit = false]
> > 9727 IFD [Lucene Merge Thread #0]: delete "_1t.fnm"
> > 9728 IFD [Lucene Merge Thread #0]: delete "_1t.frq"
> > 9729 IFD [Lucene Merge Thread #0]: delete "_1t.prx"
> > 9730 IFD [Lucene Merge Thread #0]: delete "_1t.tis"
> > 9731 IFD [Lucene Merge Thread #0]: delete "_1t.tii"
> > 9732 IFD [Lucene Merge Thread #0]: delete "_1t.nrm"
> > 9733 IFD [Lucene Merge Thread #0]: delete "_1t.fdx"
> > 9734 IFD [Lucene Merge Thread #0]: delete "_1t.fdt"
> > 9735 IW 131 [Lucene Merge Thread #0]: CMS:   merge thread: done
> > 9736 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> now
> > commit transaction
> > 9737 IFD [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]: now
> > checkpoint "segments_1u" [1 segments ; isCo      mmit = false]
> > 9738 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > startCommit(): start sizeInBytes=0
> > 9739 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > startCommit index=_1t:c262436 changeCount=4
> > 9740 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> now
> > sync _1t.cfs
> > 9741 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > done all syncs
> > 9742 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > commit: pendingCommit != null
> > 9743 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > commit: wrote segments file "segments_1v"
> > 9744 IFD [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]: now
> > checkpoint "segments_1v" [1 segments ; isCo      mmit = true]
> > 9745 IFD [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > deleteCommits: now decRef commit "segments_1u"
> > 9746 IFD [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> delete
> > "_1s.cfs"
> > 9747 IFD [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> delete
> > "segments_1u"
> > 9748 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > commit: done
> > 9749 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > LMP: findMerges: 1 segments
> > 9750 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > LMP:   level 7.2867203 to 8.03672: 1 segment      s
> > 9751 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS: now merge
> > 9752 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   index: _1t:c262436
> > 9753 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   no more merges pending; now return
> > 9754 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> now
> > flush at close
> > 9755 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: now pause all indexing threads
> > 9756 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > flush: segment=null docStoreSegment=null d      ocStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > 9757 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > index before flush _1t:c262436
> > 9758 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS: now merge
> > 9759 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   index: _1t:c262436
> > 9760 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > CMS:   no more merges pending; now return
> > 9761 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> now
> > call final commit()
> > 9762 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > startCommit(): start sizeInBytes=0
> > 9763 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > skip startCommit(): no changes pending
> > 9764 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > commit: pendingCommit == null; skip
> > 9765 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]:
> > commit: done
> > 9766 IW 131 [MC:10.10.176.148-1321053007682-361 FS:emag_393219_0]: at
> > close: _1t:c262436
> > 9767 IFD [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > setInfoStream deletionPolicy=org.apache.lucene.
> > index.KeepOnlyLastCommitDeletionPolicy@8506f9<mailto:index.KeepOnlyLas
> > t
> > CommitDeletionPolicy@8506f9>
> > 9768 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > setInfoStream: dir=org.apache.lucene.store.S
> > impleFSDirectory@/cas/fs_589826/sideline/fs_393219/search/index_0/prim
> > ary
> > autoCommit=true mergePolicy=org.
> >
> apache.lucene.index.LogByteSizeMergePolicy@18d0653<mailto:apache.lucene.
> > index.LogByteSizeMergePolicy@18d0653>
> > mergeScheduler=org.apache.lucene.index.ConcurrentMergeS
> > cheduler@19a5bf2 ramBufferSizeMB=16.0 maxBufferedDocs=-1
> > maxBuffereDeleteTerms=-1 maxFieldLength=10000 ind      ex=_1t:c262436
> > 9769 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > setMaxBufferedDocs 10000
> > 9770 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > flush at addIndexes
> > 9771 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > flush: now pause all indexing threads
> > 9772 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > flush: segment=null docStoreSegment=null d      ocStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > 9773 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > index before flush _1t:c262436
> > 9774 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> now
> > start transaction
> > 9775 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > flush at startTransaction
> > 9776 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > flush: now pause all indexing threads
> > 9777 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > flush: segment=null docStoreSegment=null d      ocStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > 9778 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > index before flush _1t:c262436
> > 9779 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > optimize: index now _1t:c262436 _10.10.176.1      48-1321045422606-
> > 204_0:cx4000**
> > 9780 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > flush: now pause all indexing threads
> > 9781 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > flush: segment=null docStoreSegment=null d      ocStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > 9782 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > index before flush _1t:c262436 _10.10.176.      148-1321045422606-
> > 204_0:cx4000**
> > 9783 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> add
> > merge to pendingMerges: _1t:c262436 _10.      10.176.148-1321045422606-
> > 204_0:cx4000 [optimize] [total 1 pending]
> > 9784 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > CMS: now merge
> > 9785 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > CMS:   index: _1t:c262436 _10.10.176.148-132
> 1045422606-204_0:cx4000**
> > 9786 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > CMS:   consider merge _1t:c262436 _10.10.176      .148-1321045422606-
> > 204_0:cx4000 into _1u [optimize] [mergeDocStores]
> > 9787 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > CMS:     launch new thread [Lucene Merge Thr      ead #0]
> > 9788 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > CMS:   no more merges pending; now return
> > 9789 IW 132 [Lucene Merge Thread #0]: CMS:   merge thread: start
> > 9790 IW 132 [Lucene Merge Thread #0]: now merge
> > 9791   merge=_1t:c262436 _10.10.176.148-1321045422606-204_0:cx4000
> into
> > _1u [optimize] [mergeDocStores]
> > 9792
> >
> merge=org.apache.lucene.index.MergePolicy$OneMerge@3b0c25<mailto:mer
> > ge=org.apache.lucene.index.MergePolicy$OneMerge@3b0c25>
> > 9793   index=_1t:c262436 _10.10.176.148-1321045422606-204_0:cx4000**
> > 9794 IW 132 [Lucene Merge Thread #0]: merging _1t:c262436
> _10.10.176.148-
> > 1321045422606-204_0:cx4000 into _1u [o      ptimize] [mergeDocStores]
> > 9795 IW 132 [Lucene Merge Thread #0]: merge: total 266436 docs
> > 9796 IW 132 [Lucene Merge Thread #0]: handleMergeException:
> > merge=_1t:c262436 _10.10.176.148-1321045422606-204_      0:cx4000 into
> > _1u [optimize] [mergeDocStores]
> > exc=org.apache.lucene.index.CorruptIndexException: docs out       of
order
> > (262439 <= 2096427 )
> > 9797 IW 132 [Lucene Merge Thread #0]: hit exception during merge
> > 9798 IFD [Lucene Merge Thread #0]: refresh [prefix=_1u]: removing
> > newly created unreferenced file "_1u.fdt"
> > 9799 IFD [Lucene Merge Thread #0]: delete "_1u.fdt"
> > 9800 IFD [Lucene Merge Thread #0]: refresh [prefix=_1u]: removing
> > newly created unreferenced file "_1u.fdx"
> > 9801 IFD [Lucene Merge Thread #0]: delete "_1u.fdx"
> > 9802 IFD [Lucene Merge Thread #0]: refresh [prefix=_1u]: removing
> > newly created unreferenced file "_1u.fnm"
> > 9803 IFD [Lucene Merge Thread #0]: delete "_1u.fnm"
> > 9804 IFD [Lucene Merge Thread #0]: refresh [prefix=_1u]: removing
> > newly created unreferenced file "_1u.frq"
> > 9805 IFD [Lucene Merge Thread #0]: delete "_1u.frq"
> > 9806 IFD [Lucene Merge Thread #0]: refresh [prefix=_1u]: removing
> > newly created unreferenced file "_1u.prx"
> > 9807 IFD [Lucene Merge Thread #0]: delete "_1u.prx"
> > 9808 IFD [Lucene Merge Thread #0]: refresh [prefix=_1u]: removing
> > newly created unreferenced file "_1u.tii"
> > 9809 IFD [Lucene Merge Thread #0]: delete "_1u.tii"
> > 9810 IFD [Lucene Merge Thread #0]: refresh [prefix=_1u]: removing
> > newly created unreferenced file "_1u.tis"
> > 9811 IFD [Lucene Merge Thread #0]: delete "_1u.tis"
> > 9812 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> now
> > rollback transaction
> > 9813 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> all
> > running merges have aborted
> > 9814 IFD [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]: now
> > checkpoint "segments_1v" [1 segments ; isCo      mmit = false]
> > 9815 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> now
> > flush at close
> > 9816 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > flush: now pause all indexing threads
> > 9817 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > flush: segment=null docStoreSegment=null d      ocStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > 9818 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > index before flush _1t:c262436
> > 9819 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > CMS: now merge
> > 9820 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > CMS:   index: _1t:c262436
> > 9821 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > CMS:   no more merges pending; now return
> > 9822 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> now
> > call final commit()
> > 9823 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > startCommit(): start sizeInBytes=0
> > 9824 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > startCommit index=_1t:c262436 changeCount=1
> > 9825 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > done all syncs
> > 9826 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > commit: pendingCommit != null
> > 9827 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > commit: wrote segments file "segments_1w"
> > 9828 IFD [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]: now
> > checkpoint "segments_1w" [1 segments ; isCo      mmit = true]
> > 9829 IFD [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > deleteCommits: now decRef commit "segments_1v"
> > 9830 IFD [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> delete
> > "segments_1v"
> > 9831 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > commit: done
> > 9832 IW 132 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]: at
> > close: _1t:c262436
> > 9833 IFD [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > setInfoStream deletionPolicy=org.apache.lucene.
> >
> index.KeepOnlyLastCommitDeletionPolicy@1ec4b8e<mailto:index.KeepOnlyLa
> > s
> > tCommitDeletionPolicy@1ec4b8e>
> > 9834 IW 133 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > setInfoStream: dir=org.apache.lucene.store.S
> > impleFSDirectory@/cas/fs_589826/sideline/fs_393219/search/index_0/prim
> > ary
> > autoCommit=true mergePolicy=org.
> > apache.lucene.index.LogByteSizeMergePolicy@546a0e<mailto:apache.lucene
> > .i
> > ndex.LogByteSizeMergePolicy@546a0e>
> > mergeScheduler=org.apache.lucene.index.ConcurrentMergeSc
> > heduler@1fef5b8 ramBufferSizeMB=16.0 maxBufferedDocs=-1
> > maxBuffereDeleteTerms=-1 maxFieldLength=10000 inde      x=_1t:c262436
> > 9835 IW 133 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > setMaxBufferedDocs 10000
> > 9836 IW 133 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > flush at addIndexes
> > 9837 IW 133 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > flush: now pause all indexing threads
> > 9838 IW 133 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > flush: segment=null docStoreSegment=null d      ocStoreOffset=0
> > flushDocs=false flushDeletes=true flushDocStores=false numDocs=0
> > numBufDelTerms=0
> > 9839 IW 133 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> > index before flush _1t:c262436
> > 9840 IW 133 [MC:10.10.176.148-1321045422606-204 FS:emag_393219_0]:
> now
> > start transaction
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message