lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: 140GB index directory, what can I do?
Date Mon, 16 Aug 2010 09:31:34 GMT
Optimizing an index every 60 seconds is a *very* bad idea! Don't do this!
Performance will dramatically decrease. Since Lucene 2.9 you don’t need to
optimize in most cases, as search is working per-segment. Optimizing an
index should only be done during maintenance or if you want to ship a static
index e.g. on a DVD. For live installations, the merge policy will manage
index segments for you fine and index reopen times and field cache for
sorting will be working fast. If you optimize every 60 seconds, you have to
reopen the whole index, redo all caches and so on.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Andrew Bruno [mailto:andrew.bruno@gmail.com]
> Sent: Monday, August 16, 2010 3:35 AM
> To: java-user
> Subject: Re: 140GB index directory, what can I do?
> 
> The index is optimized every 60 secs... so it must have already been
cleaned up.
> 
> Thanks for feedback.
> 
> On Sat, Aug 14, 2010 at 8:15 PM, Uwe Schindler <uwe@thetaphi.de> wrote:
> > Optimize the index one time then all unused segment files are
> > *for-sure* removed.
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> >> -----Original Message-----
> >> From: Andrew Bruno [mailto:andrew.bruno@gmail.com]
> >> Sent: Saturday, August 14, 2010 12:11 PM
> >> To: java-user@lucene.apache.org
> >> Subject: 140GB index directory, what can I do?
> >>
> >> Hi all,
> >>
> >> I have an index directory that is growing pretty fast, and is now at
> > 138GB.
> >>
> >> A while ago, this index got corrupted.  It was rebuilt, but the
> >> engineer
> > cannot
> >> remember whether he deleted the corrupt directory before the rebuild.
> >> Is there a way to know if any files are not being used or whether
> >> they can be purged, merged, etc?
> >>
> >> The files are listed below.
> >>
> >> 08/14/2010  02:51 AM    <DIR>          ..
> >> 03/24/2010  11:22 PM                 0 clearcache
> >> 08/14/2010  02:51 AM                 4 deletable
> >> 08/14/2010  02:51 AM               369 segments
> >> 03/29/2010  02:29 AM   135,941,156,712 _1x3.cfs
> >> 08/13/2010  08:24 AM           275,615 _1x3.del
> >> 05/10/2010  09:59 AM       977,999,933 _491b.cfs
> >> 08/10/2010  12:46 PM            17,304 _491b.del
> >> 06/18/2010  09:33 PM       809,473,470 _7sr7.cfs
> >> 08/06/2010  07:10 AM            15,038 _7sr7.del
> >> 07/23/2010  11:34 PM       559,421,479 _aeps.cfs
> >> 08/13/2010  07:20 AM            11,276 _aeps.del
> >> 08/09/2010  09:23 AM       299,717,058 _bu0l.cfs
> >> 08/12/2010  12:42 PM             6,387 _bu0l.del
> >> 08/11/2010  02:46 PM        66,910,259 _c4x4.cfs
> >> 08/11/2010  04:06 PM             1,401 _c4x4.del
> >> 08/13/2010  02:11 PM        45,009,744 _cc9c.cfs
> >> 08/13/2010  03:59 PM               944 _cc9c.del
> >> 08/13/2010  04:03 PM         3,409,701 _ccqa.cfs
> >> 08/13/2010  04:29 PM                79 _ccqa.del
> >> 08/13/2010  10:42 PM           900,521 _ccvl.cfs
> >> 08/14/2010  12:01 AM                29 _ccvl.del
> >> 08/14/2010  12:00 AM           589,015 _cczi.cfs
> >> 08/14/2010  12:07 AM                24 _cczi.del
> >> 08/14/2010  12:05 AM           260,231 _cd0v.cfs
> >> 08/14/2010  02:44 AM           324,498 _cd1z.cfs
> >> 08/14/2010  02:44 AM            10,651 _cd20.cfs
> >> 08/14/2010  02:45 AM             6,427 _cd21.cfs
> >> 08/14/2010  02:45 AM             6,702 _cd22.cfs
> >> 08/14/2010  02:45 AM             9,024 _cd23.cfs
> >> 08/14/2010  02:45 AM            10,641 _cd24.cfs
> >> 08/14/2010  02:46 AM             9,037 _cd25.cfs
> >> 08/14/2010  02:46 AM             6,297 _cd26.cfs
> >> 08/14/2010  02:46 AM             6,504 _cd27.cfs
> >> 08/14/2010  02:46 AM            56,164 _cd28.cfs
> >> 08/14/2010  02:47 AM                 9 _cd28.del
> >> 08/14/2010  02:47 AM            56,246 _cd29.cfs
> >> 08/14/2010  02:47 AM             7,093 _cd2a.cfs
> >> 08/14/2010  02:47 AM                 9 _cd2a.del
> >> 08/14/2010  02:47 AM             7,179 _cd2b.cfs
> >> 08/14/2010  02:48 AM            10,324 _cd2c.cfs
> >> 08/14/2010  02:48 AM                 9 _cd2c.del
> >> 08/14/2010  02:48 AM            10,412 _cd2d.cfs
> >> 08/14/2010  02:49 AM            67,808 _cd2e.cfs
> >> 08/14/2010  02:49 AM             6,431 _cd2f.cfs
> >> 08/14/2010  02:49 AM             6,419 _cd2g.cfs
> >> 08/14/2010  02:49 AM            10,394 _cd2h.cfs
> >> 08/14/2010  02:49 AM            10,417 _cd2i.cfs
> >> 08/14/2010  02:49 AM             8,379 _cd2j.cfs
> >> 08/14/2010  02:50 AM             5,982 _cd2k.cfs
> >> 08/14/2010  02:50 AM             5,670 _cd2l.cfs
> >> 08/14/2010  02:51 AM             5,622 _cd2m.cfs
> >>               51 File(s) 138,705,840,941 bytes
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message