lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zheng Lin Edwin Yeo <edwinye...@gmail.com>
Subject Re: Memory Usage increases by a lot during and after optimization .
Date Mon, 04 Jan 2016 02:05:45 GMT
Thanks for the reply Shawn and Erick.

What *exactly* are you looking at that says Solr is using all your
memory?  You must be extremely specific when answering this question.
This will determine whether we should be looking for a bug or not.

A) Before I start the optimization, the server's memory usage
is consistent at around 16GB, when Solr startsup and we did some searching.
However, when I click on the optimization button, the memory usage
increases gradually, until it reaches the maximum of 64GB which the server
has. But this only happens to the collection with index of 200GB, and not
other collections which has smaller index size (they are at most 1GB at the
moment).


In another message thread, you indicated that your max heap was set to
14GB.  Java will only ever use that much memory for the program that is
being run, plus a relatively small amount so that Java itself can
operate.  Any significantly large resident memory allocation beyond the
max heap would be an indication of a bug in Java, not a bug in Solr.

A) I am quite curious at this also, because in the Task Manager of the
server, the amount of memory usage stated does not tally with the
percentage of memory usage. When I start optimizatoin, the memory usage
states the JVM is only using 14GB, but the percentage of memory usage is
almost 100%, when I have 64GB RAM. I have check the other processes running
in the server, and did not found any other processes that takes up a large
amount of memory, and the total amount of memory usage for the whole sever
is only around 16GB.


Regards,
Edwin


On 3 January 2016 at 01:24, Erick Erickson <erickerickson@gmail.com> wrote:

> If you happen to be looking at "top" or the like, you
> might be seeing virtual memory, see Uwe's
> excellent blog here:
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> Best,
> Erick
>
> On Fri, Jan 1, 2016 at 11:46 PM, Shawn Heisey <apache@elyograg.org> wrote:
> > On 12/31/2015 8:03 PM, Zheng Lin Edwin Yeo wrote:
> >> But the problem I'm facing now is that during optimizing, the memory
> usage
> >> of the server hit the maximum of 64GB, and I believe the optimization
> could
> >> not be completed fully as there is not enough memory, so when I check
> the
> >> index again, it says that it is not optimized. Before the optimization,
> the
> >> memory usage was less than 16GB, so the optimization actually uses up
> more
> >> than 48GB of memory.
> >>
> >> Is it normal for an index size of 200GB to use up so much memory during
> >> optimization?
> >
> > What *exactly* are you looking at that says Solr is using all your
> > memory?  You must be extremely specific when answering this question.
> > This will determine whether we should be looking for a bug or not.
> >
> > It is completely normal for all modern operating systems to use all the
> > memory when the amount of data being handled is large.  Some of the
> > memory will be allocated to programs like Java/Solr, and the operating
> > system will use everything else to cache data from I/O operations on the
> > disk.  This is called the page cache.  For Solr to perform well, the
> > page cache must be large enough to effectively cache your index data.
> >
> > https://en.wikipedia.org/wiki/Page_cache
> >
> > In another message thread, you indicated that your max heap was set to
> > 14GB.  Java will only ever use that much memory for the program that is
> > being run, plus a relatively small amount so that Java itself can
> > operate.  Any significantly large resident memory allocation beyond the
> > max heap would be an indication of a bug in Java, not a bug in Solr.
> >
> > With the index size at 200GB, I would hope to have at least 128GB of
> > memory in the server, but I would *want* 256GB.  64GB may not be enough
> > for good performance.
> >
> > Thanks,
> > Shawn
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message