lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Memory Usage increases by a lot during and after optimization .
Date Sat, 02 Jan 2016 17:24:21 GMT
If you happen to be looking at "top" or the like, you
might be seeing virtual memory, see Uwe's
excellent blog here:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

Best,
Erick

On Fri, Jan 1, 2016 at 11:46 PM, Shawn Heisey <apache@elyograg.org> wrote:
> On 12/31/2015 8:03 PM, Zheng Lin Edwin Yeo wrote:
>> But the problem I'm facing now is that during optimizing, the memory usage
>> of the server hit the maximum of 64GB, and I believe the optimization could
>> not be completed fully as there is not enough memory, so when I check the
>> index again, it says that it is not optimized. Before the optimization, the
>> memory usage was less than 16GB, so the optimization actually uses up more
>> than 48GB of memory.
>>
>> Is it normal for an index size of 200GB to use up so much memory during
>> optimization?
>
> What *exactly* are you looking at that says Solr is using all your
> memory?  You must be extremely specific when answering this question.
> This will determine whether we should be looking for a bug or not.
>
> It is completely normal for all modern operating systems to use all the
> memory when the amount of data being handled is large.  Some of the
> memory will be allocated to programs like Java/Solr, and the operating
> system will use everything else to cache data from I/O operations on the
> disk.  This is called the page cache.  For Solr to perform well, the
> page cache must be large enough to effectively cache your index data.
>
> https://en.wikipedia.org/wiki/Page_cache
>
> In another message thread, you indicated that your max heap was set to
> 14GB.  Java will only ever use that much memory for the program that is
> being run, plus a relatively small amount so that Java itself can
> operate.  Any significantly large resident memory allocation beyond the
> max heap would be an indication of a bug in Java, not a bug in Solr.
>
> With the index size at 200GB, I would hope to have at least 128GB of
> memory in the server, but I would *want* 256GB.  64GB may not be enough
> for good performance.
>
> Thanks,
> Shawn
>

Mime
View raw message