lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bogaert <boggu...@gmail.com>
Subject Re: lots of .cfs (compound files) in the index directory
Date Thu, 15 Mar 2012 16:02:11 GMT
Mike,

while removing the prepareCommit we noticed we didn't actually called the
IW.commit() method before the IW.close().
Altough the documentation says the close() method commits all the changes
we tried to add the commit() method before the IW.close() and this seems to
fix our problem.
Is this a bug or just a misunderstanding of the documentation and you
always have to call the commit() method first?

Thanks for your help,
Tim


On Thu, Mar 15, 2012 at 4:20 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> Hmm are you calling IW.prepareCommit() before calling IW.close()?
>
> Can you try removing the call the IW.prepareCommit() and see if the
> segments are then contained...?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Thu, Mar 15, 2012 at 11:14 AM, Tim Bogaert <bogguard@gmail.com> wrote:
> > Operating system is Solaris. Directory is a NAS.
> > Directory implementation is SimpleFSDirectory.
> > I sent you the full log.
> >
> > Thanks,
> > Tim
> >
> > On Thu, Mar 15, 2012 at 4:04 PM, Michael McCandless <
> > lucene@mikemccandless.com> wrote:
> >
> >> Hmm, which OS/filesystem is the index on?  Which Directory impl are
> >> you using...?
> >>
> >> Indeed CheckIndex sees the 5111 segments, so something seems amiss.
> >>
> >> The log was edited right?  EG I can't see the IW open log entries...
> >> and IW 51 is missing (it goes straight from 50 to 52).  Can you post
> >> the full log?  Also, can you attach it (instead of inlining it), maybe
> >> gzip'd, so it doesn't get line-wrapped?
> >>
> >> Finally, can you run things a big longer, so we can see the
> >> pathological behavior emerge?
> >>
> >> The only odd thing I see in the log is that IW seems to be attempting
> >> to delete files (eg _1a.cfs) that are still "live", which is odd.
> >>
> >> That the delete did not succeed could just be that the file is open
> >> from a reader (do you have a reader open on the index?)... though, I
> >> would have expected to see "unable to remove file..." in the
> >> InfoStream output.
> >>
> >> Mike McCandless
> >>
> >> http://blog.mikemccandless.com
> >>
> >>
> >> On Thu, Mar 15, 2012 at 10:30 AM, Tim Bogaert <bogguard@gmail.com>
> wrote:
> >> > Hi Mike,
> >> >
> >> > Segment format: FORMAT_3_1 [Lucene 3.1+]
> >> > NumBadSegments: 0
> >> > NumSegments: 5111
> >> > Clean?: true
> >> > TotLoseDocCount: 0
> >> > Partial: false
> >> >
> >> >
> >> > I reindexed and let the feed come in again...The logs of the result
> are
> >> > below (I just copied a few) after 10 minutes :
> >> > Segment format: FORMAT_3_1 [Lucene 3.1+]
> >> > NumBadSegments: 0
> >> > NumSegments: 11
> >> > Clean?: true
> >> > TotLoseDocCount: 0
> >> > Partial: false
> >> >
> >> > What I noticed in the log was that _1a.cfs is getting deleted, but it
> >> seems
> >> > to be still in the directory :
> >> >
> >> > ls -al
> >> > -rw-r--r--   1 web      web      1444172 Mar 15 15:13 _17.fdt
> >> > -rw-r--r--   1 web      web        88804 Mar 15 15:13 _17.fdx
> >> > -rw-r--r--   1 web      web          142 Mar 15 15:13 _17.fnm
> >> > -rw-r--r--   1 web      web      4352933 Mar 15 15:13 _17.frq
> >> > -rw-r--r--   1 web      web       177604 Mar 15 15:13 _17.nrm
> >> > -rw-r--r--   1 web      web      5931105 Mar 15 15:13 _17.prx
> >> > -rw-r--r--   1 web      web        18159 Mar 15 15:13 _17.tii
> >> > -rw-r--r--   1 web      web      1421338 Mar 15 15:13 _17.tis
> >> > -rw-r--r--   1 web      web           36 Mar 15 15:14 _17_1.del
> >> > -rw-r--r--   1 web      web       379400 Mar 15 15:13 _18.cfs
> >> > -rw-r--r--   1 web      web       424376 Mar 15 15:13 _19.cfs
> >> > -rw-r--r--   1 web      web       463177 Mar 15 15:13 _1a.cfs
> >> > -rw-r--r--   1 web      web       108875 Mar 15 15:13 _1b.cfs
> >> > -rw-r--r--   1 web      web         4481 Mar 15 15:14 _1c.cfs
> >> > -rw-r--r--   1 web      web         4465 Mar 15 15:15 _1d.cfs
> >> > -rw-r--r--   1 web      web         4534 Mar 15 15:17 _1e.cfs
> >> > -rw-r--r--   1 web      web         3252 Mar 15 15:18 _1f.cfs
> >> > -rw-r--r--   1 web      web         3314 Mar 15 15:21 _1g.cfs
> >> > -rw-r--r--   1 web      web         2340 Mar 15 15:25 _1h.cfs
> >> > -rw-r--r--   1 web      web         3902 Mar 15 15:27 _1i.cfs
> >> > -rw-r--r--   1 web      web         1687 Mar 15 15:28 _1j.cfs
> >> > -rw-r--r--   1 web      web           20 Mar 15 15:28 segments.gen
> >> > -rw-r--r--   1 web      web         2735 Mar 15 15:28 segments_1h
> >> >
> >> >
> >> > Here is (part of) the log :
> >> >
> >> > 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #2]: TMP:
> >> > seg=_h(3.5):c1 size=0.007 MB [merging] [floored]
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #2]: TMP:
> >> > seg=_b(3.5):c1 size=0.007 MB [merging] [floored]
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #2]: TMP:
> >> > seg=_g(3.5):c1 size=0.006 MB [merging] [floored]
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #2]: TMP:
> >> > seg=_c(3.5):c1 size=0.006 MB [merging] [floored]
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #2]: TMP:
> >> > seg=_11(3.5):c1 size=0.005 MB [merging] [floored]
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #2]: TMP:
> >> > seg=_12(3.5):c1 size=0.005 MB [merging] [floored]
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #2]: TMP:
> >> > seg=_6(3.5):c1 size=0.005 MB [merging] [floored]
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #2]: TMP:
> >> > allowedSegmentCount=1 vs count=15 (eligible count=5) tooBigCount=0
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #2]: merge
> time
> >> 46
> >> > msec for 10 docs
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: now flush at
> close
> >> > waitForMerges=true
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #2]: CMS:
> >> merge
> >> > thread: done
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: now trigger
> flush
> >> > reason=explicit flush
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #2]: CMS: set
> >> > priority of merge thread Lucene Merge Thread #3 to 6
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]:   start flush:
> >> > applyAllDeletes=true
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]:   index before
> >> flush
> >> > _0(3.5):C73 _1(3.5):C70 _2(3.5):C23 _3(3.5):c7 _16(3.5):c10 _6(3.5):c1
> >> > _18(3.5):c10 _17(3.5):c10 _b(3.5):c1 _c(3.5):c1 _d(3.5):c1 _g(3.5):c1
> >> > _h(3.5):c1 _11(3.5):c1 _12(3.5):c1
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: DW: flush: no
> >> docs;
> >> > skipping
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: apply all
> deletes
> >> > during flush
> >> > BD 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: applyDeletes:
> no
> >> > deletes; skipping
> >> > BD 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: prune
> >> > sis=org.apache.lucene.index.SegmentInfos@1cad1e6 minGen=0
> packetCount=0
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]:
> clearFlushPending
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: CMS: now merge
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: CMS:   index:
> >> > _0(3.5):C73 _1(3.5):C70 _2(3.5):C23 _3(3.5):c7 _16(3.5):c10 _6(3.5):c1
> >> > _18(3.5):c10 _17(3.5):c10 _b(3.5):c1 _c(3.5):c1 _d(3.5):c1 _g(3.5):c1
> >> > _h(3.5):c1 _11(3.5):c1 _12(3.5):c1
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: CMS:   no more
> >> > merges pending; now return
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: waitForMerges
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #3]: merge
> store
> >> > matchedCount=10 vs 10
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #3]: merged
> >> > segment size=0.076 MB vs estimate=0.103 MB
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #3]:
> >> commitMerge:
> >> > _2(3.5):C23 _3(3.5):c7 _d(3.5):c1 _h(3.5):c1 _b(3.5):c1 _g(3.5):c1
> >> > _c(3.5):c1 _11(3.5):c1 _12(3.5):c1 _6(3.5):c1 into _19
> index=_0(3.5):C73
> >> > _1(3.5):C70 _2(3.5):C23 _3(3.5):c7 _16(3.5):c10 _6(3.5):c1
> _18(3.5):c10
> >> > _17(3.5):c10 _b(3.5):c1 _c(3.5):c1 _d(3.5):c1 _g(3.5):c1 _h(3.5):c1
> >> > _11(3.5):c1 _12(3.5):c1
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #3]:
> >> > commitMergeDeletes _2(3.5):C23 _3(3.5):c7 _d(3.5):c1 _h(3.5):c1
> >> _b(3.5):c1
> >> > _g(3.5):c1 _c(3.5):c1 _11(3.5):c1 _12(3.5):c1 _6(3.5):c1 into _19
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #3]: after
> >> commit:
> >> > _0(3.5):C73 _1(3.5):C70 _19(3.5):C38 _16(3.5):c10 _18(3.5):c10
> >> _17(3.5):c10
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #3]: now
> >> checkpoint
> >> > "segments_15" [6 segments ; isCommit = false]
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #3]: merge
> time
> >> 61
> >> > msec for 38 docs
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: waitForMerges
> done
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; Lucene Merge Thread #3]: CMS:
> >> merge
> >> > thread: done
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: now call final
> >> > commit()
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: commit: start
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: commit: enter
> lock
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: commit: already
> >> > prepared
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: commit:
> >> > pendingCommit != null
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: commit: wrote
> >> > segments file "segments_16"
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: now checkpoint
> >> > "segments_16" [42 segments ; isCommit = true]
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: deleteCommits:
> now
> >> > decRef commit "segments_15"
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: delete
> "segments_15"
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: commit: done
> >> > IW 50 [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: at close:
> >> > _0(3.5):C73 _1(3.5):C70 _19(3.5):C38 _16(3.5):c10 _18(3.5):c10
> >> _17(3.5):c10
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: delete "_19.tii"
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: delete "_19.tis"
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: delete "_18.cfs"
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: delete "_19.nrm"
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: delete "_19.fnm"
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: delete "_19.fdt"
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: delete "_17.cfs"
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: delete "_19.prx"
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: delete "_16.cfs"
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: delete "_19.fdx"
> >> > IFD [Thu Mar 15 15:25:23 MET 2012; pool-2-thread-1]: delete "_19.frq"
> >> > Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: applyDeletes: no
> deletes;
> >> > skipping
> >> > BD 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: prune
> >> > sis=org.apache.lucene.index.SegmentInfos@59d9fe minGen=0
> packetCount=0
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]:
> clearFlushPending
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: CMS: now merge
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: CMS:   index:
> >> > _0(3.5):C73 _1(3.5):C70 _2(3.5):C23 _1a(3.5):C16 _17(3.5):c10
> >> _19(3.5):c10
> >> > _18(3.5):c10
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: CMS:   no more
> >> > merges pending; now return
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: waitForMerges
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: waitForMerges
> done
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: now call final
> >> > commit()
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: commit: start
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: commit: enter
> lock
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: commit: already
> >> > prepared
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: commit:
> >> > pendingCommit != null
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: commit: wrote
> >> > segments file "segments_17"
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: now checkpoint
> >> > "segments_17" [43 segments ; isCommit = true]
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: deleteCommits:
> now
> >> > decRef commit "segments_16"
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: delete
> "segments_16"
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: commit: done
> >> > IW 52 [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: at close:
> >> > _0(3.5):C73 _1(3.5):C70 _2(3.5):C23 _1a(3.5):C16 _17(3.5):c10
> >> _19(3.5):c10
> >> > _18(3.5):c10
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: delete "_1a.fdx"
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: delete "_1a.tis"
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: delete "_1a.frq"
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: delete "_1a.tii"
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: delete "_1a.fnm"
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: delete "_1a.fdt"
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: delete "_18.cfs"
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: delete "_1a.nrm"
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: delete "_19.cfs"
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: delete "_17.cfs"
> >> > IFD [Thu Mar 15 15:25:27 MET 2012; pool-2-thread-1]: delete "_1a.prx"
> >> >
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; Lucene Merge Thread #3]: CMS:
> >> merge
> >> > thread: done
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; Lucene Merge Thread #0]: CMS: set
> >> > priority of merge thread Lucene Merge Thread #3 to 6
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: now flush at
> close
> >> > waitForMerges=true
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: now trigger
> flush
> >> > reason=explicit flush
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]:   start flush:
> >> > applyAllDeletes=true
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]:   index before
> >> flush
> >> > _0(3.5):C73 _1(3.5):C70 _2(3.5):C23 _3(3.5):c7 _18(3.5):c10
> _1b(3.5):c10
> >> > _1a(3.5):c10 _19(3.5):c10
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: DW: flush: no
> >> docs;
> >> > skipping
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: apply all
> deletes
> >> > during flush
> >> > BD 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: applyDeletes:
> no
> >> > deletes; skipping
> >> > BD 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: prune
> >> > sis=org.apache.lucene.index.SegmentInfos@e91b11 minGen=0
> packetCount=0
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]:
> clearFlushPending
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: CMS: now merge
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: CMS:   index:
> >> > _0(3.5):C73 _1(3.5):C70 _2(3.5):C23 _3(3.5):c7 _18(3.5):c10
> _1b(3.5):c10
> >> > _1a(3.5):c10 _19(3.5):c10
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: CMS:   no more
> >> > merges pending; now return
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: waitForMerges
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: waitForMerges
> done
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: now call final
> >> > commit()
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: commit: start
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: commit: enter
> lock
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: commit: already
> >> > prepared
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: commit:
> >> > pendingCommit != null
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: commit: wrote
> >> > segments file "segments_18"
> >> > IFD [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: now checkpoint
> >> > "segments_18" [44 segments ; isCommit = true]
> >> > IFD [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: deleteCommits:
> now
> >> > decRef commit "segments_17"
> >> > IFD [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: delete
> "segments_17"
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: commit: done
> >> > IW 53 [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: at close:
> >> > _0(3.5):C73 _1(3.5):C70 _2(3.5):C23 _3(3.5):c7 _18(3.5):c10
> _1b(3.5):c10
> >> > _1a(3.5):c10 _19(3.5):c10
> >> > IFD [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: delete "_1b.cfs"
> >> > IFD [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: delete "_18.cfs"
> >> > IFD [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: delete "_19.cfs"
> >> > IFD [Thu Mar 15 15:25:38 MET 2012; pool-2-thread-1]: delete "_1a.cfs"
> >> >
> >> > Kind regards,
> >> > Bogguard
> >> >
> >> > On Thu, Mar 15, 2012 at 1:21 PM, Michael McCandless <
> >> > lucene@mikemccandless.com> wrote:
> >> >
> >> >> Hmm, that's odd...
> >> >>
> >> >> Can you set IndexWriter's infoStream and then capture the output
> while
> >> >> doing the small writes every few seconds and post back?
> >> >>
> >> >> If you run CheckIndex on the index does it also report ~3000
> segments?
> >> >>
> >> >> Mike McCandless
> >> >>
> >> >> http://blog.mikemccandless.com
> >> >>
> >> >> On Thu, Mar 15, 2012 at 7:14 AM, Tim Bogaert <bogguard@gmail.com>
> >> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > We migrated our project to Lucene 3.5 and used the
> TieredMergePolicy
> >> with
> >> >> > all its defaults.
> >> >> > Our project contains a relatively small index (+-90MB) and gets
> small
> >> >> > writes every few seconds.
> >> >> > The behaviour we see is that when we reindex everything works
great
> >> and
> >> >> > performant,
> >> >> > but when we keep the index running for some days we see a lot
> >> (+-3000) of
> >> >> > very small .cfs files (between 1KB and 10KB) are being generated
> and
> >> the
> >> >> > performance drops significantly.
> >> >> > Reïndexing solves the problem, but this shouldn't be the solution.
> >> >> >
> >> >> > Is there a way to change the behaviour of the .cfs files. I noticed
> >> you
> >> >> can
> >> >> > fully disable it, but I read that this isn't adviced.
> >> >> >
> >> >> > Kind regards,
> >> >> > Bogguard
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >> >>
> >> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message