Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 78637 invoked from network); 14 Apr 2009 09:29:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 14 Apr 2009 09:29:08 -0000 Received: (qmail 41065 invoked by uid 500); 14 Apr 2009 09:29:06 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 41010 invoked by uid 500); 14 Apr 2009 09:29:05 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 41000 invoked by uid 99); 14 Apr 2009 09:29:05 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Apr 2009 09:29:05 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [87.233.177.133] (HELO core.aduna-software.com) (87.233.177.133) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Apr 2009 09:28:55 +0000 Received: from localhost (localhost [127.0.0.1]) by core.aduna-software.com (Postfix) with ESMTP id 69E62128001 for ; Tue, 14 Apr 2009 11:27:41 +0200 (CEST) X-Virus-Scanned: amavisd-new at X-Spam-Score: -2.573 X-Spam-Level: Received: from core.aduna-software.com ([127.0.0.1]) by localhost (core.aduna-software.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aWAxUtWSNhQE for ; Tue, 14 Apr 2009 11:27:36 +0200 (CEST) Received: from [10.1.0.11] (5350580E.static.ziggozakelijk.nl [83.80.88.14]) by core.aduna-software.com (Postfix) with ESMTP id B9401AA037A for ; Tue, 14 Apr 2009 11:27:36 +0200 (CEST) Message-ID: <49E4573E.7070208@aduna-software.com> Date: Tue, 14 Apr 2009 11:28:30 +0200 From: Christiaan Fluit User-Agent: Thunderbird 2.0.0.21 (Windows/20090302) MIME-Version: 1.0 To: java-user@lucene.apache.org Subject: semi-infinite loop during merging Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Flag: NO X-Old-Spam-Status: No, score=-2.573 tagged_above=-10 required=4 tests=[AWL=0.026, BAYES_00=-2.599] Hello all, I have a very peculiar problem that is driving me crazy: on some of our datasets and at some point in time during indexing, the merge operation runs into a (semi-)infinite loop and keeps adding files to the index until it runs out of free disk space. The situation: I have an indexing application that uses Lucene 2.4.1. Only one IndexWriter is involved, operating on a FSDirectory and using the compound file format. The index is created from scratch. No IndexReaders or IndexSearchers are open during indexing (double-checked by adding explicit log statements where they are created). For reasons unrelated to Lucene, the application is compiled with JET, a commercial Java Windows compiler. A regular Java build has produced the problem only once. The JET build does it every time - unless I keep pressing F5 continuously in Windows Explorer on the index dir. Here is what I see happening in the index dir: - At first, it builds _0.cfs to _9.cfs without problems. The files vary in size between 12 MB and 49 MB and add up to about 250 MB. - Then, .del files are generated for some of these .cfs files. The number of .del files and the .cfs files they correspond with differs from time to time. I don't understand why these are created, as no IndexReaders exist at this time. - Then, it generates files called _a.fdt, _a.fdx, _a.fnm, _a.frq, _a.nrm, _a.prx, _a.tii, _a.tis, _a.tvd, _a.tvf, _a.tvx. Together these files add up to 219 MB. I assume this is the start of the merge of the 10 .cfs files and this is all still correct. - Then, it generates _b files with those same extensions, then _c, _d, etc. It only keeps generating new files, I never see files disappear. The original .cfs files are still there. - This continues until my hard drive is out of free space. At one test I was at _8n and the index had grown from 250MB to 64 GB. Then the application just hangs. Interestingly, after killing the application in this test, there were _8k.cfs and _8m.cfs files of 20 MB and 27 MB respectively. No other .cfs files existed. In some older threads on this list (e.g. http://marc.info/?l=lucene-user&m=108300530413241&w=2) I read that "Win32 seems to sometimes not permit one to delete a file immediately after it has been closed". Could this explain the problem? Perhaps the JET-compiled app gets to delete the file quicker than when the code is running inside a Java VM and therefore runs into this issue? This would also explain why pressing F5 during indexing lets the application continue: external activity causing some manual delay. At the end of this mail I have added the output of the InfoStream installed on the IndexWriter, showing everything from the start to the first few problematic merges. Lines starting with "===" are println's in my own code to make sure that indeed only IndexWriters are generated and no IndexSearchers/-Readers. The fun starts almost at the bottom, at the first line containing "LMP: findMerges: 10 segments". The next lines then get repeated over and over again, with different segment names. I cannot explain why it mentions "1 deleted docIDs" a couple of lines below the first "findMerges: 10" as no IndexWriter.deleteDocuments takes place. As you can see in this output, I am setting a SerialMergeScheduler to rule out concurrency issues and making debugging easier. Both SerialMergeScheduler and ConcurrentMergeScheduler give this problem though. I would be grateful if anyone could pose some light on this or can advise me on what I can try next. I even considered hacking the FSDirectory code and adding some delay in the deleteFile operation to see if the above-mentioned win32 issue is the problem, but unless you know what you're doing, such hacks can even cause such problems in the first place. Kind regards, Chris -- === Creating new FSDirectory === Opening indexWriter (create=true) IFD [AWT-EventQueue-0]: setInfoStream deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@9fb0688 IW 0 [AWT-EventQueue-0]: setInfoStream: dir=org.apache.lucene.store.FSDirectory@D:\index autoCommit=false mergePolicy=org.apache.lucene.index.LogByteSizeMergePolicy@a11d0d8 mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@a3fa9d8 ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 maxFieldLength=2147483647 index= IW 0 [AWT-EventQueue-0]: setMergeScheduler org.apache.lucene.index.SerialMergeScheduler@a3d1fe8 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=46.015 vs trigger=16 allocMB=46.015 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=46.015 allocMB=46.015 IW 0 [CrawlThread]: flush: segment=_0 docStoreSegment=_0 docStoreOffset=0 flushDocs=true flushDeletes=false flushDocStores=false numDocs=2193 numBufDelTerms=0 IW 0 [CrawlThread]: index before flush IW 0 [CrawlThread]: DW: flush postings as segment _0 numDocs=2193 IW 0 [CrawlThread]: DW: oldRAMSize=48249856 newFlushedSize=13379990 docs/MB=171.863 new/old=27.731% IFD [CrawlThread]: now checkpoint "segments_1" [1 segments ; isCommit = false] IFD [CrawlThread]: now checkpoint "segments_1" [1 segments ; isCommit = false] IFD [CrawlThread]: delete "_0.fnm" IFD [CrawlThread]: delete "_0.frq" IFD [CrawlThread]: delete "_0.prx" IFD [CrawlThread]: delete "_0.tis" IFD [CrawlThread]: delete "_0.tii" IFD [CrawlThread]: delete "_0.nrm" IW 0 [CrawlThread]: LMP: findMerges: 1 segments IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 1 segments IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs trigger=16 allocMB=46.015 vs trigger=16.8 byteBlockFree=12.438 charBlockFree=8.969 IW 0 [CrawlThread]: DW: after free: freedMB=30.841 usedMB=0.044 allocMB=15.173 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=6.806 vs trigger=16 allocMB=16.829 vs trigger=16.8 byteBlockFree=0.812 charBlockFree=0 IW 0 [CrawlThread]: DW: after free: freedMB=1.656 usedMB=6.806 allocMB=15.173 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=9.242 vs trigger=16 allocMB=16.861 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: after free: freedMB=1.687 usedMB=9.242 allocMB=15.174 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=43.339 vs trigger=16 allocMB=43.401 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=43.339 allocMB=43.401 IW 0 [CrawlThread]: flush: segment=_1 docStoreSegment=_0 docStoreOffset=2193 flushDocs=true flushDeletes=false flushDocStores=false numDocs=1799 numBufDelTerms=0 IW 0 [CrawlThread]: index before flush _0:c2193->_0 IW 0 [CrawlThread]: DW: flush postings as segment _1 numDocs=1799 IW 0 [CrawlThread]: DW: oldRAMSize=45444096 newFlushedSize=12691373 docs/MB=148.635 new/old=27.927% IFD [CrawlThread]: now checkpoint "segments_1" [2 segments ; isCommit = false] IFD [CrawlThread]: now checkpoint "segments_1" [2 segments ; isCommit = false] IFD [CrawlThread]: delete "_1.fnm" IFD [CrawlThread]: delete "_1.frq" IFD [CrawlThread]: delete "_1.prx" IFD [CrawlThread]: delete "_1.tis" IFD [CrawlThread]: delete "_1.tii" IFD [CrawlThread]: delete "_1.nrm" IW 0 [CrawlThread]: LMP: findMerges: 2 segments IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 2 segments IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs trigger=16 allocMB=43.401 vs trigger=16.8 byteBlockFree=11.781 charBlockFree=8.469 IW 0 [CrawlThread]: DW: after free: freedMB=28.217 usedMB=0.044 allocMB=15.185 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=40.889 vs trigger=16 allocMB=40.951 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=40.889 allocMB=40.951 IW 0 [CrawlThread]: flush: segment=_2 docStoreSegment=_0 docStoreOffset=3992 flushDocs=true flushDeletes=false flushDocStores=false numDocs=1013 numBufDelTerms=0 IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 IW 0 [CrawlThread]: DW: flush postings as segment _2 numDocs=1013 IW 0 [CrawlThread]: DW: oldRAMSize=42874880 newFlushedSize=11684845 docs/MB=90.905 new/old=27.253% IFD [CrawlThread]: now checkpoint "segments_1" [3 segments ; isCommit = false] IFD [CrawlThread]: now checkpoint "segments_1" [3 segments ; isCommit = false] IFD [CrawlThread]: delete "_2.fnm" IFD [CrawlThread]: delete "_2.frq" IFD [CrawlThread]: delete "_2.prx" IFD [CrawlThread]: delete "_2.tis" IFD [CrawlThread]: delete "_2.tii" IFD [CrawlThread]: delete "_2.nrm" IW 0 [CrawlThread]: LMP: findMerges: 3 segments IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 3 segments IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs trigger=16 allocMB=40.951 vs trigger=16.8 byteBlockFree=10.688 charBlockFree=8.062 IW 0 [CrawlThread]: DW: after free: freedMB=25.779 usedMB=0.044 allocMB=15.172 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=38.82 vs trigger=16 allocMB=38.883 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=38.82 allocMB=38.883 IW 0 [CrawlThread]: flush: segment=_3 docStoreSegment=_0 docStoreOffset=5005 flushDocs=true flushDeletes=false flushDocStores=false numDocs=427 numBufDelTerms=0 IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 IW 0 [CrawlThread]: DW: flush postings as segment _3 numDocs=427 IW 0 [CrawlThread]: DW: oldRAMSize=40706048 newFlushedSize=10574425 docs/MB=42.342 new/old=25.978% IFD [CrawlThread]: now checkpoint "segments_1" [4 segments ; isCommit = false] IFD [CrawlThread]: now checkpoint "segments_1" [4 segments ; isCommit = false] IFD [CrawlThread]: delete "_3.fnm" IFD [CrawlThread]: delete "_3.frq" IFD [CrawlThread]: delete "_3.prx" IFD [CrawlThread]: delete "_3.tis" IFD [CrawlThread]: delete "_3.tii" IFD [CrawlThread]: delete "_3.nrm" IW 0 [CrawlThread]: LMP: findMerges: 4 segments IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 4 segments IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs trigger=16 allocMB=38.883 vs trigger=16.8 byteBlockFree=9.312 charBlockFree=7.812 IW 0 [CrawlThread]: DW: after free: freedMB=23.686 usedMB=0.044 allocMB=15.197 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=8.588 vs trigger=16 allocMB=16.822 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=8.588 allocMB=15.197 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=16.15 vs trigger=16 allocMB=19.51 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull IW 0 [CrawlThread]: DW: after free: freedMB=3.297 usedMB=16.15 allocMB=16.213 IW 0 [CrawlThread]: flush: segment=_4 docStoreSegment=_0 docStoreOffset=5432 flushDocs=true flushDeletes=false flushDocStores=false numDocs=1145 numBufDelTerms=0 IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 IW 0 [CrawlThread]: DW: flush postings as segment _4 numDocs=1145 IW 0 [CrawlThread]: DW: oldRAMSize=16934912 newFlushedSize=5040651 docs/MB=238.187 new/old=29.765% IFD [CrawlThread]: now checkpoint "segments_1" [5 segments ; isCommit = false] IFD [CrawlThread]: now checkpoint "segments_1" [5 segments ; isCommit = false] IFD [CrawlThread]: delete "_4.fnm" IFD [CrawlThread]: delete "_4.frq" IFD [CrawlThread]: delete "_4.prx" IFD [CrawlThread]: delete "_4.tis" IFD [CrawlThread]: delete "_4.tii" IFD [CrawlThread]: delete "_4.nrm" IW 0 [CrawlThread]: LMP: findMerges: 5 segments IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 5 segments IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=45.533 vs trigger=16 allocMB=45.596 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=45.533 allocMB=45.596 IW 0 [CrawlThread]: flush: segment=_5 docStoreSegment=_0 docStoreOffset=6577 flushDocs=true flushDeletes=false flushDocStores=false numDocs=808 numBufDelTerms=0 IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 IW 0 [CrawlThread]: DW: flush postings as segment _5 numDocs=808 IW 0 [CrawlThread]: DW: oldRAMSize=47745024 newFlushedSize=12761396 docs/MB=66.392 new/old=26.728% IFD [CrawlThread]: now checkpoint "segments_1" [6 segments ; isCommit = false] IFD [CrawlThread]: now checkpoint "segments_1" [6 segments ; isCommit = false] IFD [CrawlThread]: delete "_5.fnm" IFD [CrawlThread]: delete "_5.frq" IFD [CrawlThread]: delete "_5.prx" IFD [CrawlThread]: delete "_5.tis" IFD [CrawlThread]: delete "_5.tii" IFD [CrawlThread]: delete "_5.nrm" IW 0 [CrawlThread]: LMP: findMerges: 6 segments IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 6 segments IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs trigger=16 allocMB=45.596 vs trigger=16.8 byteBlockFree=11.5 charBlockFree=9.031 IW 0 [CrawlThread]: DW: after free: freedMB=30.404 usedMB=0.044 allocMB=15.192 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=7.051 vs trigger=16 allocMB=16.848 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: after free: freedMB=1.656 usedMB=7.051 allocMB=15.192 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=9.829 vs trigger=16 allocMB=16.942 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: after free: freedMB=1.75 usedMB=9.829 allocMB=15.192 IW 0 [CrawlThread]: hit exception adding document IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=48.622 vs trigger=16 allocMB=48.685 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=48.622 allocMB=48.685 IW 0 [CrawlThread]: flush: segment=_6 docStoreSegment=_0 docStoreOffset=7385 flushDocs=true flushDeletes=false flushDocStores=false numDocs=1689 numBufDelTerms=0 IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 IW 0 [CrawlThread]: DW: flush postings as segment _6 numDocs=1689 IW 0 [CrawlThread]: DW: oldRAMSize=50983936 newFlushedSize=14095657 docs/MB=125.645 new/old=27.647% IFD [CrawlThread]: now checkpoint "segments_1" [7 segments ; isCommit = false] IFD [CrawlThread]: now checkpoint "segments_1" [7 segments ; isCommit = false] IFD [CrawlThread]: delete "_6.fnm" IFD [CrawlThread]: delete "_6.frq" IFD [CrawlThread]: delete "_6.prx" IFD [CrawlThread]: delete "_6.tis" IFD [CrawlThread]: delete "_6.tii" IFD [CrawlThread]: delete "_6.nrm" IW 0 [CrawlThread]: LMP: findMerges: 7 segments IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 7 segments IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs trigger=16 allocMB=48.685 vs trigger=16.8 byteBlockFree=12.969 charBlockFree=9.594 IW 0 [CrawlThread]: DW: after free: freedMB=33.498 usedMB=0.044 allocMB=15.187 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=7.007 vs trigger=16 allocMB=16.812 vs trigger=16.8 byteBlockFree=0.281 charBlockFree=0 IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=7.007 allocMB=15.187 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=9.254 vs trigger=16 allocMB=16.812 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=9.254 allocMB=15.188 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=11.913 vs trigger=16 allocMB=16.844 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: after free: freedMB=1.656 usedMB=11.913 allocMB=15.188 IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=14.566 vs trigger=16 allocMB=16.813 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=14.566 allocMB=15.189 IW 0 [CrawlThread]: DW: RAM: now flush @ usedMB=16.151 allocMB=16.214 triggerMB=16 IW 0 [CrawlThread]: flush: segment=_7 docStoreSegment=_0 docStoreOffset=9074 flushDocs=true flushDeletes=false flushDocStores=false numDocs=2941 numBufDelTerms=0 IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 IW 0 [CrawlThread]: DW: flush postings as segment _7 numDocs=2941 IW 0 [CrawlThread]: DW: oldRAMSize=16935936 newFlushedSize=5920901 docs/MB=520.843 new/old=34.961% IFD [CrawlThread]: now checkpoint "segments_1" [8 segments ; isCommit = false] IFD [CrawlThread]: now checkpoint "segments_1" [8 segments ; isCommit = false] IFD [CrawlThread]: delete "_7.fnm" IFD [CrawlThread]: delete "_7.frq" IFD [CrawlThread]: delete "_7.prx" IFD [CrawlThread]: delete "_7.tis" IFD [CrawlThread]: delete "_7.tii" IFD [CrawlThread]: delete "_7.nrm" IW 0 [CrawlThread]: LMP: findMerges: 8 segments IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 8 segments IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=14.513 vs trigger=16 allocMB=16.808 vs trigger=16.8 byteBlockFree=0 charBlockFree=0.75 IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=14.513 allocMB=15.183 IW 0 [CrawlThread]: DW: RAM: now flush @ usedMB=16.057 allocMB=16.277 triggerMB=16 IW 0 [CrawlThread]: flush: segment=_8 docStoreSegment=_0 docStoreOffset=12015 flushDocs=true flushDeletes=false flushDocStores=false numDocs=3171 numBufDelTerms=0 IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 IW 0 [CrawlThread]: DW: flush postings as segment _8 numDocs=3171 IW 0 [CrawlThread]: DW: oldRAMSize=16836608 newFlushedSize=7011110 docs/MB=474.252 new/old=41.642% IFD [CrawlThread]: now checkpoint "segments_1" [9 segments ; isCommit = false] IFD [CrawlThread]: now checkpoint "segments_1" [9 segments ; isCommit = false] IFD [CrawlThread]: delete "_8.fnm" IFD [CrawlThread]: delete "_8.frq" IFD [CrawlThread]: delete "_8.prx" IFD [CrawlThread]: delete "_8.tis" IFD [CrawlThread]: delete "_8.tii" IFD [CrawlThread]: delete "_8.nrm" IW 0 [CrawlThread]: LMP: findMerges: 9 segments IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 9 segments IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=15.678 vs trigger=16 allocMB=17.178 vs trigger=16.8 byteBlockFree=1.438 charBlockFree=0 IW 0 [CrawlThread]: DW: nothing to free IW 0 [CrawlThread]: DW: after free: freedMB=1.438 usedMB=15.678 allocMB=15.74 IW 0 [CrawlThread]: DW: RAM: now flush @ usedMB=16.067 allocMB=16.13 triggerMB=16 IW 0 [CrawlThread]: flush: segment=_9 docStoreSegment=_0 docStoreOffset=15186 flushDocs=true flushDeletes=false flushDocStores=false numDocs=3401 numBufDelTerms=0 IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 IW 0 [CrawlThread]: DW: flush postings as segment _9 numDocs=3401 IW 0 [CrawlThread]: DW: oldRAMSize=16847872 newFlushedSize=6184635 docs/MB=576.624 new/old=36.709% IFD [CrawlThread]: now checkpoint "segments_1" [10 segments ; isCommit = false] IFD [CrawlThread]: now checkpoint "segments_1" [10 segments ; isCommit = false] IFD [CrawlThread]: delete "_9.fnm" IFD [CrawlThread]: delete "_9.frq" IFD [CrawlThread]: delete "_9.prx" IFD [CrawlThread]: delete "_9.tis" IFD [CrawlThread]: delete "_9.tii" IFD [CrawlThread]: delete "_9.nrm" IW 0 [CrawlThread]: LMP: findMerges: 10 segments IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments IW 0 [CrawlThread]: LMP: 0 to 10: add this merge IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 [total 1 pending] IW 0 [CrawlThread]: DW: apply 0 buffered deleted terms and 1 deleted docIDs and 0 deleted queries on 10 segments. IFD [CrawlThread]: now checkpoint "segments_1" [10 segments ; isCommit = false] IW 0 [CrawlThread]: now flush at merge IW 0 [CrawlThread]: flush: segment=null docStoreSegment=_0 docStoreOffset=18587 flushDocs=false flushDeletes=false flushDocStores=true numDocs=0 numBufDelTerms=0 IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 IW 0 [CrawlThread]: flush shared docStore segment _0 IW 0 [CrawlThread]: DW: closeDocStore: 2 files to flush to segment _0 numDocs=18587 IFD [CrawlThread]: now checkpoint "segments_1" [10 segments ; isCommit = false] IFD [CrawlThread]: delete "_0.fdt" IFD [CrawlThread]: delete "_0.fdx" IFD [CrawlThread]: delete "_0.fdt" IFD [CrawlThread]: delete "_0.fdx" IW 0 [CrawlThread]: now merge merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _a merge=org.apache.lucene.index.MergePolicy$OneMerge@b590b98 index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 IW 0 [CrawlThread]: merging _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _a IW 0 [CrawlThread]: merge: total 18586 docs IW 0 [CrawlThread]: LMP: findMerges: 10 segments IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments IW 0 [CrawlThread]: LMP: 0 to 10: add this merge IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 [total 1 pending] IW 0 [CrawlThread]: now merge merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _b merge=org.apache.lucene.index.MergePolicy$OneMerge@b590658 index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 IW 0 [CrawlThread]: merging _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _b IW 0 [CrawlThread]: merge: total 18586 docs IW 0 [CrawlThread]: LMP: findMerges: 10 segments IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments IW 0 [CrawlThread]: LMP: 0 to 10: add this merge IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 [total 1 pending] IW 0 [CrawlThread]: now merge merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _c merge=org.apache.lucene.index.MergePolicy$OneMerge@b5906b8 index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 IW 0 [CrawlThread]: merging _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _c IW 0 [CrawlThread]: merge: total 18586 docs IW 0 [CrawlThread]: LMP: findMerges: 10 segments IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments IW 0 [CrawlThread]: LMP: 0 to 10: add this merge IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 [total 1 pending] IW 0 [CrawlThread]: now merge merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _d merge=org.apache.lucene.index.MergePolicy$OneMerge@b5906e8 index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 IW 0 [CrawlThread]: merging _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _d IW 0 [CrawlThread]: merge: total 18586 docs IW 0 [CrawlThread]: LMP: findMerges: 10 segments IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments IW 0 [CrawlThread]: LMP: 0 to 10: add this merge IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 [total 1 pending] IW 0 [CrawlThread]: now merge merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _e merge=org.apache.lucene.index.MergePolicy$OneMerge@b590688 index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 ....... --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org