lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless" <luc...@mikemccandless.com>
Subject Re: Urgent : How much actually the disk space needed to optimize the index?
Date Tue, 13 Mar 2007 09:20:10 GMT

"maureen tanuwidjaja" <autumn_musique@yahoo.com> wrote:

>   How much actually the disk space needed to optimize the index?The 
>   explanation given in documentation seems to be very different with the 
>   practical situation
>   
>   I have an index file of size 18.6 G and I am going to optimize it.I 
>   keep this index in mobile Hard Disk with capacity of 100 Gb....I did 
>   not use any index reader,and I merely call index writer to optimize 
>   this index.However,to my surprise,now while optimizing, the Index size 
>   grow to almost occupy all the free space.I am preety sure that later it
>    will terminated due to there is no sufficient disk space.
>   
>   This is the content on the index file
>   ------------------------------------------------------------------------------------------
>   03/13/2007  02:14 PM    <DIR>          .
>   03/13/2007  02:14 PM    <DIR>          ..
>   03/13/2007  02:14  PM                 20 segments.gen
>   03/13/2007  02:14  PM                 67 segments_34s4
>   03/13/2007  12:06  PM                  0 write.lock
>   03/13/2007  02:14 PM    41,705,009,152 _1ke1.cfs
>   03/13/2007  12:15 PM     1,638,320,227 _1ke1.fdt
>   03/13/2007  12:15 PM         4,461,912 _1ke1.fdx
>   03/13/2007  12:09 PM         6,295,065 _1ke1.fnm
>   03/13/2007  12:26 PM       232,520,666 _1ke1.frq
>   03/13/2007  02:08 PM    44,927,549,671 _1ke1.nrm
>   03/13/2007  12:26 PM       170,766,513 _1ke1.prx
>   03/13/2007  12:26 PM         1,281,924 _1ke1.tii
>   03/13/2007  12:26 PM       103,094,835 _1ke1.tis
>   03/13/2007  02:14 PM        51,688,575 _1ke1.tvd
>   03/13/2007  02:14 PM       882,304,866 _1ke1.tvf
>   03/13/2007  02:14 PM         4,461,916 _1ke1.tvx
>   03/12/2007  03:24 PM     5,594,336,501 _8km.cfs


As best I know, it should only require 2X the disk space.  In your
case this means you should only have needed 18.6 GB of free space (ie,
1X is the current index, then another 1X in free space).

So something odd is happening here.

One thing that stands out in your listing is: your norms file
(_1ke1.nrm) is enormous compared to all other files.  Are you indexing
many tiny docs where each docs has highly variable fields or something?

Hmmm.  In fact if you are doing this, then on merge, the norms (which
are not stored "sparsely") could in fact grow far larger than 2X.

Can you send a listing of the 18.6 GB index before optimizing?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message