lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Solr Memory Usage
Date Wed, 29 Oct 2014 18:13:53 GMT
On 10/29/2014 11:43 AM, Vijay Kokatnur wrote:
> I am observing some weird behavior with how Solr is using memory.  We are
> running both Solr and zookeeper on the same node.  We tested memory
> settings on Solr Cloud Setup of 1 shard with 146GB index size, and 2 Shard
> Solr setup with 44GB index size.  Both are running on similar beefy
> machines.
>
>  After running the setup for 3-4 days, I see that a lot of memory is
> inactive in all the nodes -
>
>  99052952  total memory
>  98606256  used memory
>  19143796  active memory
>  75063504  inactive memory
>
> And inactive memory is never reclaimed by the OS.  When total memory size
> is reached, latency and disk IO shoots up.  We observed this behavior in
> both Solr Cloud setup with 1 shard and Solr setup with 2 shards.

Where are these numbers coming from?  If they are coming from the
operating system and not Java, then you have nothing to worry about.

> For the Solr Cloud setup, we are running a cron job with following command
> to clear out the inactive memory.  It  is working as expected.  Even though
> the index size of Cloud is 146GB, the used memory is always below 55GB.
> Our response times are better and no errors/exceptions are thrown. (This
> command causes issue in 2 Shard setup)
>
> echo 3 > /proc/sys/vm/drop_caches

Don't do that.  You're throwing away almost every performance advantage
the operating system has to offer.  If this changes the numbers so they
look better to you, then I can almost guarantee you that you are not
having any actual problem, and that dropping the caches like this is
*hurting* performance, not helping it.

It's completely normal for a correctly functioning system to report an
extremely low amount of memory as free.  The operating system is using
the spare memory in your system as a filesystem cache, which makes
everything run a lot faster.  If a program needs more memory, the
operating system will instantly give up some of its disk cache in order
to satisfy the memory allocation.

The "virtual memory" part of this blog post (which has direct relevance
for Solr) hopefully can explain it better than I can.  The entire blog
post is worth reading.

http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

Thanks,
Shawn


Mime
View raw message