lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Memory Usage increases by a lot during and after optimization .
Date Wed, 06 Jan 2016 20:42:37 GMT
On 1/5/2016 11:50 PM, Zheng Lin Edwin Yeo wrote:
> Here is the new screenshot of the Memory tab of the Resource Monitor.
> https://www.dropbox.com/s/w4bnrb66r16lpx1/Resource%20Monitor.png?dl=0
>
> Yes, I found that the value under the "Working Set" column is much higher
> than the others. Also, the value which I was previously looking at under
> the Task Manager is under the Private Column here.
> It says that I have about 14GB of available memory, but the "Free" number
> is much lower, at 79MB.

You'll probably think I'm nuts, but I believe everything is working
exactly as it should.

The first two processes, which I assume are Solr processes, show a
Shareable size near 7GB each.  I have seen something similar happen on
Linux where SHR memory is huge for the Solr process, and when this
happens, the combination of memory numbers would turn out to be
impossible, so I think it's a memory reporting bug related to Java, one
that affects both Linux and Windows.

Subtracting SHR from RES (or in your case, Shareable from Working)
reveals the actual memory being used, and I believe you can see this
actual number in the Private column, which is approximately the
difference between Working and Shareable.  If I'm right, this means that
the actual memory usage is almost 14GB lower than Windows is reporting.

If both of the top processes are Solr, I'm not sure why you have two
Solr processes on one machine.  One Solr instance can handle multiple
indexes with no problem.

As evidence that I'm not insane, consider the following screenshot, from
another of my servers:

https://www.dropbox.com/s/64en3sar4cr1ytj/linux-solr-mem-high-shr.png?dl=0

On the screenshot, the solr process shows RES size of 22GB ... which is
highly unusual, because this Solr install has a max heap of 8GB ... but
notice that SHR is 13GB.  The difference between 22GB and 13GB is 9GB,
which is much more reasonable, and if we assume that the 22GB is rounded
up and/or the 13GB is rounded down, then the difference is much closer
to 8GB.  Looking at some other numbers, the "cached" value is 48GB.  If
you add the 48GB cache allocation to the *reported* resident size of
22GB for Solr, you get a total of 70GB ... which is more memory than the
machine even has (64GB).  This is why I am sure that when SHR is really
high on a Java process, it is a memory reporting error.

Thanks,
Shawn


Mime
View raw message