lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Index partition corrupted during a regular flush due to FileNotFoundException on DEL file
Date Tue, 13 Sep 2016 12:49:07 GMT
On 9/12/2016 7:07 PM, 郑文兴 wrote:
> So you mean if there is no more than 10G free space, Lucene/Solr will
> delete some files to save the disk space? Or it will cause the
> misbehave of Lucene/Solr?

If you do not have enough free disk space, and a segment merge takes
place that requires more free space than you have, then the merge will
fail.  Merges are going to happen if you are adding/updating documents
in your index.

The Solr operation called "optimize" is known as "forceMerge" in
Lucene.  It forces a merge of the entire index down to (usually) one
segment.  Here's some more information:

https://lucene.apache.org/core/5_3_2/core/org/apache/lucene/index/IndexWriter.html#forceMerge(int)

Lucene will not delete files to free up space.  The only way that a
Lucene index ever gets smaller is through segment merging, which will
temporarily *increase* the space used, until the merge is complete.  Any
deleted documents contained in the merging segments will be removed when
the merge completes.

> Please note that we have several shards/partitions under the same root
> directory, so which way the following is true to us. Let's assume we
> have 2 partitions, and A -> 10G, B->10G
>
> l  do we have to make sure there are at least 20G disk space available?
>
> l  Or we just need to make sure there are at least 10G disk space
> available?
>

The extra space required for merging is detailed in the link above.  At
worst your index size might temporarily increase by a factor of four. 
Usually a full optimize (forceMerge) on an index that's not in the
compound file format will only *double* the index size while it's
running, but in some situations, it might require more.

My own recommendation:  If you only have one large index on a server,
you should have enough space available for that index to triple in
size.  If you have many large indexes, having enough space for *all*
your indexes to double in size at the same time is probably sufficient,
but if you can arrange for more free space, that would be advisable.

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message