lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zheng Lin Edwin Yeo <edwinye...@gmail.com>
Subject Re: Memory Usage increases by a lot during and after optimization .
Date Thu, 07 Jan 2016 07:53:01 GMT
Hi Shawn,

Thank you for your explanation.

Yes, both of the top two processes are Solr. I have two Solr processes on
one machine now, as the second one is a replica of the first one. In the
future, the plan is to have them on separate machine.


>Subtracting SHR from RES (or in your case, Shareable from Working)
>reveals the actual memory being used, and I believe you can see this
>actual number in the Private column, which is approximately the
>difference between Working and Shareable.  If I'm right, this means that
>the actual memory usage is almost 14GB lower than Windows is reporting.

Does this means that sometimes when I see the high memory usage (can be up
to 100%), it is just a memory reporting error by Windows, but Solr is
working exactly as it should?

Regards,
Edwin


On 7 January 2016 at 04:42, Shawn Heisey <apache@elyograg.org> wrote:

> On 1/5/2016 11:50 PM, Zheng Lin Edwin Yeo wrote:
> > Here is the new screenshot of the Memory tab of the Resource Monitor.
> > https://www.dropbox.com/s/w4bnrb66r16lpx1/Resource%20Monitor.png?dl=0
> >
> > Yes, I found that the value under the "Working Set" column is much higher
> > than the others. Also, the value which I was previously looking at under
> > the Task Manager is under the Private Column here.
> > It says that I have about 14GB of available memory, but the "Free" number
> > is much lower, at 79MB.
>
> You'll probably think I'm nuts, but I believe everything is working
> exactly as it should.
>
> The first two processes, which I assume are Solr processes, show a
> Shareable size near 7GB each.  I have seen something similar happen on
> Linux where SHR memory is huge for the Solr process, and when this
> happens, the combination of memory numbers would turn out to be
> impossible, so I think it's a memory reporting bug related to Java, one
> that affects both Linux and Windows.
>
> Subtracting SHR from RES (or in your case, Shareable from Working)
> reveals the actual memory being used, and I believe you can see this
> actual number in the Private column, which is approximately the
> difference between Working and Shareable.  If I'm right, this means that
> the actual memory usage is almost 14GB lower than Windows is reporting.
>
> If both of the top processes are Solr, I'm not sure why you have two
> Solr processes on one machine.  One Solr instance can handle multiple
> indexes with no problem.
>
> As evidence that I'm not insane, consider the following screenshot, from
> another of my servers:
>
> https://www.dropbox.com/s/64en3sar4cr1ytj/linux-solr-mem-high-shr.png?dl=0
>
> On the screenshot, the solr process shows RES size of 22GB ... which is
> highly unusual, because this Solr install has a max heap of 8GB ... but
> notice that SHR is 13GB.  The difference between 22GB and 13GB is 9GB,
> which is much more reasonable, and if we assume that the 22GB is rounded
> up and/or the 13GB is rounded down, then the difference is much closer
> to 8GB.  Looking at some other numbers, the "cached" value is 48GB.  If
> you add the 48GB cache allocation to the *reported* resident size of
> 22GB for Solr, you get a total of 70GB ... which is more memory than the
> machine even has (64GB).  This is why I am sure that when SHR is really
> high on a Java process, it is a memory reporting error.
>
> Thanks,
> Shawn
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message