lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: optimization problem
Date Tue, 17 Feb 2009 12:43:17 GMT

It's odd that optimize is creating such tiny segments, and then that  
these tiny segments wind up consuming so much disk space.

Can you turn on IndexWriter's infoStream, and post the output of the  
attempts to optimize?

Are you sure there are no unhandled exceptions being logged to the  
system error log?  By default, merges are done in BG threads (using  
ConcurrentMergeScheduler), so exceptions hit in those threads may be  
logged elsewhere (though optimize() tries to bring such exceptions to  
the foreground thread).

Mike

Qingdi wrote:

>
> (I posted this question to "solr user" forum, but didn't get a clear  
> answer.
> So re-post it here.)
>
> Our index size is about 60G. Most of the time, the optimization  
> works fine.
> But this morning, the optimization kept creating new segment files  
> until all
> the free disk space (300G) was used up.
>
> Here is how the files generated during optimization look like:
> ============================================
> ....................................
> -rw-r--r--  1 root root      213464 Feb 10 07:33 _5w9a.tis
> -rw-r--r--  1 root root        3039 Feb 10 07:33 _5w9a.tii
> -rw-r--r--  1 root root        9360 Feb 10 07:33 _5w9a.prx
> -rw-r--r--  1 root root       19824 Feb 10 07:33 _5w9a.frq
> -rw-r--r--  1 root root          27 Feb 10 07:33 _5w9a.fnm
> -rw-r--r--  1 root root       74884 Feb 10 07:33 _5w9a.fdx
> -rw-r--r--  1 root root     3753944 Feb 10 07:33 _5w9a.fdt
> -rw-r--r--  1 root root      212513 Feb 10 07:33 _5w99.tis
> -rw-r--r--  1 root root        3005 Feb 10 07:33 _5w99.tii
> -rw-r--r--  1 root root        9436 Feb 10 07:33 _5w99.prx
> -rw-r--r--  1 root root       20052 Feb 10 07:33 _5w99.frq
> -rw-r--r--  1 root root          27 Feb 10 07:33 _5w99.fnm
> -rw-r--r--  1 root root       75492 Feb 10 07:33 _5w99.fdx
> -rw-r--r--  1 root root     3795439 Feb 10 07:33 _5w99.fdt
> -rw-r--r--  1 root root      215726 Feb 10 07:33 _5w98.tis
> -rw-r--r--  1 root root        2994 Feb 10 07:33 _5w98.tii
> -rw-r--r--  1 root root        9438 Feb 10 07:33 _5w98.prx
> -rw-r--r--  1 root root       20058 Feb 10 07:33 _5w98.frq
> -rw-r--r--  1 root root          27 Feb 10 07:33 _5w98.fnm
> -rw-r--r--  1 root root       75508 Feb 10 07:33 _5w98.fdx
> -rw-r--r--  1 root root     3779402 Feb 10 07:33 _5w98.fdt
> ....................................
> ============================================
>
> This is the third time we encountered this issue. Any idea why the
> optimization keeps creating new files? (If we restart the solr  
> server and
> run optimization again, the index files can be recovered. we are on  
> solr
> 1.3.) Where could we find more info. about how the optimization  
> process
> work?
>
> Appreciate your help.
>
> Qingdi
> -- 
> View this message in context: http://www.nabble.com/optimization-problem-tp21966759p21966759.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message