lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avishai Ish-Shalom <avis...@fewbytes.com>
Subject Re: Solr memory usage off-heap
Date Tue, 18 Mar 2014 14:23:13 GMT
aha! mmap explains it. thank you.


On Tue, Mar 18, 2014 at 3:11 PM, Shawn Heisey <solr@elyograg.org> wrote:

> On 3/18/2014 5:30 AM, Avishai Ish-Shalom wrote:
> > My solr instances are configured with 10GB heap (Xmx) but linux shows
> > resident size of 16-20GB. even with thread stack and permgen taken into
> > account i'm still far off from these numbers. Could it be that jvm IO
> > buffers take so much space? does lucene use JNI/JNA memory allocations?
>
> Solr does not do anything off-heap.  There is a project called
> heliosearch underway that aims to use off-heap memory extensively with
> Solr.
>
> There IS some mis-reporting of memory usage, though.  See a screenshot
> that I just captured of top output, sorted by memory usage.  The java
> process at the top of the list is Solr, running under the included Jetty:
>
> https://www.dropbox.com/s/03a3pp510mrtixo/solr-ram-usage-wrong.png
>
> I have a 6GB heap and 52GB of index data on this server.  This makes the
> 62.2GB virtual memory size completely reasonable.  The claimed resident
> memory size is 20GB, though.  If you add that 20GB to the 49GB that is
> allocated to the OS disk cache and the 6GB that it says is free, that's
> 75GB.  I've only got 64GB of RAM on the box, so something is being
> reported wrong.
>
> If I take my 20GB resident size and subtract the 14GB shared size, that
> is closer to reality, and it makes the numbers fit into the actual
> amount of RAM that's on the machine.  I believe the misreporting is
> caused by the specific way that Java uses MMap when opening Lucene
> indexes.  This information comes from what I remember about a
> conversation I witnessed in #lucene or #lucene-dev, not from my own
> exploration.  I believe they said that the MMap methods which don't
> misreport memory usage would not do what Lucene requires.
>
> Thanks,
> Shawn
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message