lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: Solr configuration, memory usage and MMapDirectory
Date Wed, 08 Oct 2014 20:09:10 GMT
On 10/8/2014 4:02 AM, Simon Fairey wrote:
> I'm currently setting up jconsole but as I have to remotely monitor (no gui capability
on the server) I have to wait before I can restart solr with a JMX port setup. In the meantime
I looked at top and given the calculations you said based on your top output and this top
of my java process from the node that handles the querying, the indexing node has a similar
memory profile:
> 
> https://www.dropbox.com/s/pz85dm4e7qpepco/SolrTop.png?dl=0
> 
> It would seem I need a monstrously large heap in the 60GB region?
> 
> We do use a lot of navigators/filters so I have set the caches to be quite large for
these, are these what are using up the memory?

With a VIRT size of 189GB and a RES size of 73GB, I believe you probably
have more than 45GB of index data.  This might be a combination of old
indexes and the active index.  Only the indexes (cores) that are being
actively used need to be considered when trying to calculate the total
RAM needed.  Other indexes will not affect performance, even though they
increase your virtual memory size.

With MMap, part of the virtual memory size is the size of the index data
that has been opened on the disk.  This is not memory that's actually
allocated.  There's a very good reason that mmap has been the default in
Lucene and Solr for more than two years.

http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

You stated originally that you have 25 million document and 45GB of
index data on each node.  With those numbers and a conservative
configuration, I would expect that you need about 4GB of heap, maybe as
much as 8GB.  I cannot think of any reason that you would NEED a heap
60GB or larger.

Each field that you sort on, each field that you facet on with the
default facet.method of fc, and each filter that you cache will use a
large block of memory.  The size of that block of memory is almost
exclusively determined by the number of documents in the index.

With 25 million documents, each filterCache entry will be approximately
3MB -- one bit for every document.  I do not know how big each
FieldCache entry is for a sort field and a facet field, but assume that
they are probably larger than the 3MB entries on the filterCache.

I've got a filterCache sized at 64, with an autowarmCount of 4.  With
larger autowarmCount values, I was seeing commits take 30 seconds or
more, because each of those filters can take a few seconds to execute.
Cache sizes in the thousands are rarely necessary, and just chew up a
lot of memory with no benefit.  Large autowarmCount values are also
rarely necessary.  Every time a new searcher is opened by a commit, add
up all your autowarmCount values and realize that the searcher likely
needs to execute that many queries before it is available.

If you need to set up remote JMX so you can remotely connect jconsole, I
have done this in the redhat init script I've built -- see JMX_OPTS here:

http://wiki.apache.org/solr/ShawnHeisey#Init_script

It's never a good idea to expose Solr directly to the internet, but if
you use that JMX config, *definitely* don't expose it to the Internet.
It doesn't use any authentication.

We might need to back up a little bit and start with the problem that
you are trying to figure out, not the numbers that are being reported.

http://people.apache.org/~hossman/#xyproblem

Your original note said that you're sanity checking.  Toward that end,
the only insane thing that jumps out at me is that your max heap is
*VERY* large, and you probably don't have the proper GC tuning.

My recommendations for initial action are to use -Xmx8g on the servlet
container startup and include the GC settings you can find on the wiki
pages I've given you.  It would be a very good idea to set up remote JMX
so you can use jconsole or jvisualvm remotely.

Thanks,
Shawn


Mime
View raw message