lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avishai Ish-Shalom <>
Subject Re: Solr memory usage off-heap
Date Tue, 18 Mar 2014 14:23:13 GMT
aha! mmap explains it. thank you.

On Tue, Mar 18, 2014 at 3:11 PM, Shawn Heisey <> wrote:

> On 3/18/2014 5:30 AM, Avishai Ish-Shalom wrote:
> > My solr instances are configured with 10GB heap (Xmx) but linux shows
> > resident size of 16-20GB. even with thread stack and permgen taken into
> > account i'm still far off from these numbers. Could it be that jvm IO
> > buffers take so much space? does lucene use JNI/JNA memory allocations?
> Solr does not do anything off-heap.  There is a project called
> heliosearch underway that aims to use off-heap memory extensively with
> Solr.
> There IS some mis-reporting of memory usage, though.  See a screenshot
> that I just captured of top output, sorted by memory usage.  The java
> process at the top of the list is Solr, running under the included Jetty:
> I have a 6GB heap and 52GB of index data on this server.  This makes the
> 62.2GB virtual memory size completely reasonable.  The claimed resident
> memory size is 20GB, though.  If you add that 20GB to the 49GB that is
> allocated to the OS disk cache and the 6GB that it says is free, that's
> 75GB.  I've only got 64GB of RAM on the box, so something is being
> reported wrong.
> If I take my 20GB resident size and subtract the 14GB shared size, that
> is closer to reality, and it makes the numbers fit into the actual
> amount of RAM that's on the machine.  I believe the misreporting is
> caused by the specific way that Java uses MMap when opening Lucene
> indexes.  This information comes from what I remember about a
> conversation I witnessed in #lucene or #lucene-dev, not from my own
> exploration.  I believe they said that the MMap methods which don't
> misreport memory usage would not do what Lucene requires.
> Thanks,
> Shawn

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message