lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ron Mayer <r...@0ape.com>
Subject Can I tell Solr to merge *oldest* rather than smallest segments - if so I think I wouldn't need optimize anymore.
Date Fri, 13 Aug 2010 18:32:17 GMT
Short summary:

 * If I could make Solr merge oldest segments (or the one
   with the most deleted docs) rather than smallest
   segments; I think I'd almost never need "optimize".

 * Can I tell Solr to do this?  Or if not, can someone
   point me in the right direction regarding where I might
   patch it to try this myself?


I have a system where documents are refreshed and/or expired
pretty much in a FIFO manner.  In particular, no document
in the system can live for over 1 month.

Without frequent optimizes, ISTM my indexes tend to get
bloated with mostly deleted content.   I attached a ls-l
below - showing the largest segments in my index are all
from July.   A query of
   timestamp:([1999-01-01T00:00:00Z TO 2010-08-01T23:59:59Z])
returns no documents so it appears to me the first 2 segments
are entirely filled with deleted documents.

I imagine this is not too uncommon a situation -- for example
a web-crawler that periodically updates web pages that contain
some dynamic content.

Perhaps a different good criteria would be selecting to merge
the segments with the largest number of deleted documents.
In my case it'd be the same; but I could imagine non-FIFO
update-heavy systems where that would work better.




$ ls -lrt *.fdt
-rw-rw-r-- 1 ramayer ramayer 291490823897 Jul 20 21:34 _u63.fdt
-rw-rw-r-- 1 ramayer ramayer  78251326159 Jul 29 18:15 _xkh.fdt
-rw-rw-r-- 1 ramayer ramayer  69295141685 Aug  8 01:29 _10f5.fdt
-rw-rw-r-- 1 ramayer ramayer   5406369697 Aug 10 21:14 _13fv.fdt
-rw-rw-r-- 1 ramayer ramayer  66210508029 Aug 10 21:44 _13g1.fdt
-rw-rw-r-- 1 ramayer ramayer   2001873014 Aug 10 23:05 _13io.fdt
-rw-rw-r-- 1 ramayer ramayer   1578531820 Aug 11 14:10 _13m8.fdt
-rw-rw-r-- 1 ramayer ramayer   2254917604 Aug 12 03:49 _13p3.fdt
-rw-rw-r-- 1 ramayer ramayer   2890967852 Aug 12 06:49 _13s6.fdt
-rw-rw-r-- 1 ramayer ramayer   2820285238 Aug 12 09:49 _13v9.fdt
-rw-rw-r-- 1 ramayer ramayer   2905550377 Aug 12 12:52 _13yc.fdt
-rw-rw-r-- 1 ramayer ramayer   2776837514 Aug 12 15:54 _141f.fdt
-rw-rw-r-- 1 ramayer ramayer    259698816 Aug 12 16:15 _141p.fdt
-rw-rw-r-- 1 ramayer ramayer    290083173 Aug 12 16:34 _1420.fdt
-rw-rw-r-- 1 ramayer ramayer    279500106 Aug 12 16:54 _142b.fdt
-rw-rw-r-- 1 ramayer ramayer    277156197 Aug 12 17:17 _142m.fdt
-rw-rw-r-- 1 ramayer ramayer     91360010 Aug 13 00:27 _142x.fdt
-rw-rw-r-- 1 ramayer ramayer      7351514 Aug 13 00:37 _142y.fdt
-rw-rw-r-- 1 ramayer ramayer         7286 Aug 13 00:38 _142z.fdt
-rw-rw-r-- 1 ramayer ramayer           21 Aug 13 01:07 _1430.fdt
-rw-rw-r-- 1 ramayer ramayer           21 Aug 13 02:07 _1431.fdt
-rw-rw-r-- 1 ramayer ramayer           21 Aug 13 03:07 _1432.fdt
-rw-rw-r-- 1 ramayer ramayer           21 Aug 13 04:07 _1433.fdt
-rw-rw-r-- 1 ramayer ramayer      2388369 Aug 13 04:35 _1434.fdt
-rw-rw-r-- 1 ramayer ramayer           21 Aug 13 05:07 _1435.fdt
-rw-rw-r-- 1 ramayer ramayer           21 Aug 13 06:07 _1436.fdt
-rw-rw-r-- 1 ramayer ramayer           21 Aug 13 07:07 _1437.fdt
-rw-rw-r-- 1 ramayer ramayer           21 Aug 13 08:07 _1438.fdt
-rw-rw-r-- 1 ramayer ramayer           21 Aug 13 09:07 _1439.fdt
-rw-rw-r-- 1 ramayer ramayer           21 Aug 13 10:07 _143a.fdt
-rw-rw-r-- 1 ramayer ramayer       198581 Aug 13 11:04 _143b.fdt


Mime
View raw message