lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Solr RSIZE memory overusage
Date Tue, 11 Sep 2018 18:43:44 GMT
On 9/11/2018 12:14 PM, Boris Pasko wrote:
>> Run top, press shift-M to sort by memory usage, then grab a
> atop: http://oi68.tinypic.com/10pokkk.jpg
> top: http://oi63.tinypic.com/msbpfp.jpg

Looking at the second one:

The SHR value is showing 90GB.

Your Java process is in actuality only using in the ballpark of 9GB 
memory -- the difference between RES and SHR.  I have no idea why the 
SHR value goes so high sometimes, but it does.  This is a strange memory 
reporting anomaly encountered when using Java software.  The problem 
might be in the OS, or it might be in Java ... but there IS a reporting 
problem.  It does seem related to MMap, though.  You said you're using 
NRTCachingDirectoryFactory ... which *does* use MMap for all index file 
access.

Looks like the amount of memory in the machine is too large for "top" to 
display the numbers correctly.  See the "+" sign for total memory, 
buff/cache/ and avail Mem.

If we switch over to atop (a program that I did not know about) for a 
moment, you'll see that there is 113GB used by the disk cache.  So 
having 101GB (the RSIZE value for the Java process) is simply not 
possible.  That number is being reported incorrectly.

It does look like you've got in the neighborhood of 600GB of index 
data.  Only 113GB of that data is cached.  For some use cases, this will 
be plenty of cached data for good performance. For others, it won't be 
anywhere near enough.  For *perfect* performance, there will be enough 
memory for ALL of the index data to fit into memory.

Thanks,
Shawn


Mime
View raw message