lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stu Hood" <stuh...@mailtrust.com>
Subject FW: Re: Occasional Hang in IndexWriter.close()
Date Mon, 21 Apr 2008 21:24:43 GMT
Hey gang,

The finally block was not hiding any exceptions: I'm still seeing the hangs in IndexWriter.close().
I'm appending the relevant output from 'setInfoStream(System.out)'. It looks like after I
order the close, the IndexWriter begins a flush, and then hangs.

Any ideas?

Thanks,
Stu


>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>


IFD [main]: setInfoStream deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@18bbc5a
IW 0 [main]: setInfoStream: dir=org.apache.lucene.store.FSDirectory@/data1/solr/solr_dovecot1208772000-1208808000_f0cad01b1cb2966a/index
autoCommit=true mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@facf0b mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@2f0df1
ramBufferSizeMB=16.0 maxBuffereDocs=-1 maxBuffereDeleteTerms=-1 maxFieldLength=10000 index=_dn:C1349678
_j6:C1149270 _j5:C249194 _m4:C242243 _mf:C224430 _mk:C70616 _ml:C88 _mp:C50000 _mo:C9905
IW 0 [main]: setMergeScheduler org.apache.lucene.index.SerialMergeScheduler@13c6a22
>> Fetching hdfs://hadoop.sat.mlsrvr.com:54310/user/hadoopuser/indexed/dovecot/1208807515-1208808145-1208808184/part-00004
into a RAMDirectory:
>> .. done. Size: 22053888
>> Merging directories to solr_dovecot1208772000-1208808000_f0cad01b1cb2966a (factor:
4):
IW 0 [main]: flush at addIndexesNoOptimize
IW 0 [main]:   flush: segment=null docStoreSegment=null docStoreOffset=0 flushDocs=false flushDeletes=false
flushDocStores=false numDocs=0 numBufDelTerms=0
IW 0 [main]:   index before flush _dn:C1349678 _j6:C1149270 _j5:C249194 _m4:C242243 _mf:C224430
_mk:C70616 _ml:C88 _mp:C50000 _mo:C9905
IW 0 [main]: now start transaction
IW 0 [main]: flush at startTransaction
IW 0 [main]:   flush: segment=null docStoreSegment=null docStoreOffset=0 flushDocs=false flushDeletes=false
flushDocStores=false numDocs=0 numBufDelTerms=0
IW 0 [main]:   index before flush _dn:C1349678 _j6:C1149270 _j5:C249194 _m4:C242243 _mf:C224430
_mk:C70616 _ml:C88 _mp:C50000 _mo:C9905
IW 0 [main]: LMP: findMerges: 20 segments
IW 0 [main]: LMP:   level 13.601146 to 14.351146: 2 segments
IW 0 [main]: LMP:   level 12.390487 to 13.140487: 3 segments
IW 0 [main]: LMP:   level 11.542906 to 12.292906: 3 segments
IW 0 [main]: LMP:   level 10.656786 to 11.406786: 2 segments
IW 0 [main]: LMP:   level -1.0 to 8.768208: 10 segments
IW 0 [main]: LMP:     10 to 14: add this merge
IW 0 [main]: LMP:     14 to 18: add this merge
IW 0 [main]: add merge to pendingMerges: _1f:Cx1000->_0 _1g:Cx1000->_0 _1h:Cx1000->_0
_1i:Cx1000->_0 [total 1 pending]
IW 0 [main]: add merge to pendingMerges: _1j:Cx1000->_0 _1k:Cx1000->_0 _1l:Cx1000->_0
_1m:Cx1000->_0 [total 2 pending]
IW 0 [main]: now merge
 merge=_1f:Cx1000->_0 _1g:Cx1000->_0 _1h:Cx1000->_0 _1i:Cx1000->_0 into _mq
 index=_dn:C1349678 _j6:C1149270 _j5:C249194 _m4:C242243 _mf:C224430 _mk:C70616 _ml:C88 _mp:C50000
_mo:C9905 _1e:Cx50000->_0 _1f:Cx1000->_0 _1g:Cx1000->_0 _1h:Cx1000->_0 _1i:Cx1000->_0
_1j:Cx1000->_0 _1k:Cx1000->_0 _1l:Cx1000->_0 _1m:Cx1000->_0 _1n:Cx1000->_0
_1o:Cx857->_0
IW 0 [main]: merging _1f:Cx1000->_0 _1g:Cx1000->_0 _1h:Cx1000->_0 _1i:Cx1000->_0
into _mq
IW 0 [main]: merge: total 4000 docs
IW 0 [main]: commitMerge _1f:Cx1000->_0 _1g:Cx1000->_0 _1h:Cx1000->_0 _1i:Cx1000->_0
into _mq
IFD [main]: now checkpoint "segments_m" [17 segments ; isCommit = false]
IW 0 [main]: LMP: findMerges: 17 segments
IW 0 [main]: LMP:   level 13.601146 to 14.351146: 2 segments
IW 0 [main]: LMP:   level 12.390487 to 13.140487: 3 segments
IW 0 [main]: LMP:   level 11.542906 to 12.292906: 3 segments
IW 0 [main]: LMP:   level 10.656786 to 11.406786: 2 segments
IW 0 [main]: LMP:   level -1.0 to 10.283832: 7 segments
IW 0 [main]: LMP:     10 to 14: add this merge
IW 0 [main]: now merge
 merge=_1j:Cx1000->_0 _1k:Cx1000->_0 _1l:Cx1000->_0 _1m:Cx1000->_0 into _mr
 index=_dn:C1349678 _j6:C1149270 _j5:C249194 _m4:C242243 _mf:C224430 _mk:C70616 _ml:C88 _mp:C50000
_mo:C9905 _1e:Cx50000->_0 _mq:C4000 _1j:Cx1000->_0 _1k:Cx1000->_0 _1l:Cx1000->_0
_1m:Cx1000->_0 _1n:Cx1000->_0 _1o:Cx857->_0
IW 0 [main]: merging _1j:Cx1000->_0 _1k:Cx1000->_0 _1l:Cx1000->_0 _1m:Cx1000->_0
into _mr
IW 0 [main]: merge: total 4000 docs
IW 0 [main]: commitMerge _1j:Cx1000->_0 _1k:Cx1000->_0 _1l:Cx1000->_0 _1m:Cx1000->_0
into _mr
IFD [main]: now checkpoint "segments_m" [14 segments ; isCommit = false]
IW 0 [main]: LMP: findMerges: 14 segments
IW 0 [main]: LMP:   level 13.601146 to 14.351146: 2 segments
IW 0 [main]: LMP:   level 12.390487 to 13.140487: 3 segments
IW 0 [main]: LMP:   level 11.542906 to 12.292906: 3 segments
IW 0 [main]: LMP:   level 10.656786 to 11.406786: 2 segments
IW 0 [main]: LMP:   level -1.0 to 10.284627: 4 segments
IW 0 [main]: LMP:     10 to 14: add this merge
IW 0 [main]: add merge to pendingMerges: _mq:C4000 _mr:C4000 _1n:Cx1000->_0 _1o:Cx857->_0
[total 1 pending]
IW 0 [main]: now merge
 merge=_mq:C4000 _mr:C4000 _1n:Cx1000->_0 _1o:Cx857->_0 into _ms
 index=_dn:C1349678 _j6:C1149270 _j5:C249194 _m4:C242243 _mf:C224430 _mk:C70616 _ml:C88 _mp:C50000
_mo:C9905 _1e:Cx50000->_0 _mq:C4000 _mr:C4000 _1n:Cx1000->_0 _1o:Cx857->_0
IW 0 [main]: merging _mq:C4000 _mr:C4000 _1n:Cx1000->_0 _1o:Cx857->_0 into _ms
IW 0 [main]: merge: total 9857 docs
IW 0 [main]: commitMerge _mq:C4000 _mr:C4000 _1n:Cx1000->_0 _1o:Cx857->_0 into _ms
IFD [main]: now checkpoint "segments_m" [11 segments ; isCommit = false]
IFD [main]: delete "_mq.fnm"
IFD [main]: delete "_mq.frq"
IFD [main]: delete "_mq.prx"
IFD [main]: delete "_mq.tis"
IFD [main]: delete "_mq.tii"
IFD [main]: delete "_mq.nrm"
IFD [main]: delete "_mq.fdx"
IFD [main]: delete "_mq.fdt"
IFD [main]: delete "_mr.fnm"
IFD [main]: delete "_mr.frq"
IFD [main]: delete "_mr.prx"
IFD [main]: delete "_mr.tis"
IFD [main]: delete "_mr.tii"
IFD [main]: delete "_mr.nrm"
IFD [main]: delete "_mr.fdx"
IFD [main]: delete "_mr.fdt"
IW 0 [main]: LMP: findMerges: 11 segments
IW 0 [main]: LMP:   level 13.601146 to 14.351146: 2 segments
IW 0 [main]: LMP:   level 12.390487 to 13.140487: 3 segments
IW 0 [main]: LMP:   level 11.542906 to 12.292906: 3 segments
IW 0 [main]: LMP:   level 10.656786 to 11.406786: 3 segments
IW 0 [main]: add merge to pendingMerges: _1e:Cx50000->_0 [total 1 pending]
IW 0 [main]: now merge
 merge=_1e:Cx50000->_0 into _mt
 index=_dn:C1349678 _j6:C1149270 _j5:C249194 _m4:C242243 _mf:C224430 _mk:C70616 _ml:C88 _mp:C50000
_mo:C9905 _1e:Cx50000->_0 _ms:C9857
IW 0 [main]: merging _1e:Cx50000->_0 into _mt
IW 0 [main]: merge: total 50000 docs
IW 0 [main]: commitMerge _1e:Cx50000->_0 into _mt
IFD [main]: now checkpoint "segments_m" [11 segments ; isCommit = false]
IW 0 [main]: LMP: findMerges: 11 segments
IW 0 [main]: LMP:   level 13.601146 to 14.351146: 2 segments
IW 0 [main]: LMP:   level 12.390487 to 13.140487: 3 segments
IW 0 [main]: LMP:   level 11.542906 to 12.292906: 5 segments
IW 0 [main]: LMP:     5 to 9: add this merge
IW 0 [main]: LMP:   level 10.339036 to 10.916957: 1 segments
IW 0 [main]: add merge to pendingMerges: _mk:C70616 _ml:C88 _mp:C50000 _mo:C9905 [total 1
pending]
IW 0 [main]: now commit transaction
IW 0 [main]: checkpoint: wrote segments file "segments_n"
IFD [main]: now checkpoint "segments_n" [11 segments ; isCommit = true]
IFD [main]: deleteCommits: now remove commit "segments_m"
IFD [main]: delete "segments_m"
>> ... done.
>> Closing the target index...
IW 0 [main]: now flush at close
IW 0 [main]:   flush: segment=null docStoreSegment=null docStoreOffset=0 flushDocs=false flushDeletes=false
flushDocStores=false numDocs=0 numBufDelTerms=0
IW 0 [main]:   index before flush _dn:C1349678 _j6:C1149270 _j5:C249194 _m4:C242243 _mf:C224430
_mk:C70616 _ml:C88 _mp:C50000 _mo:C9905 _mt:C50000 _ms:C9857



>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>


-----Original Message-----
From: Stu Hood <stuhood@mailtrust.com>
Sent: Sunday, April 20, 2008 3:41pm
To: java-user@lucene.apache.org
Subject: Re: Occasional Hang in IndexWriter.close()

Hey Mike,

Thanks for your input... the 'IndexWriter.close' call was actually in a 'finally' block around
the merge code, without a 'catch', which I realized may have been hiding the exception (I
didn't realize close would block if an exception had occurred).

I've moved the close out of the finally block, so hopefully if an exception was being hidden,
I'll see it and report back.

Thanks,
Stu



-----Original Message-----
From: Michael McCandless <lucene@mikemccandless.com>
Sent: Saturday, April 19, 2008 4:40am
To: java-user@lucene.apache.org, stuhood@mailtrust.com
Subject: Re: Occasional Hang in IndexWriter.close()




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message