Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Subject: RE: Solr memory usage
From: =?utf-8?Q?Markus_Jelsma?= <markus.jelsma@openindex.io>
To: =?utf-8?Q?solr-user=40lucene=2Eapache=2Eorg?=
  <solr-user@lucene.apache.org>
Date: Wed, 9 Dec 2015 15:18:08 +0000
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Message-Id: <zarafa.56684630.5e3b.6a21cef40291d728@mail.openindex.io>

Steven - this fluctuation is normal, it is eating memory when documents are indexed or when searches are handled, this makes the meter go up. The garbage collector then frees the memory again. You can start to worry if there is a lot of activity but no fluctuation.

M.
=20
-----Original message-----
> From:Steven White <swhite4141@gmail.com>
> Sent: Wednesday 9th December 2015 15:56
> To: solr-user@lucene.apache.org
> Subject: Re: Solr memory usage
>=20
> Thanks Erick!!  Your summary and the blog by Uwe (thank you too Uwe) are
> very helpful.
>=20
> A follow up question.  I also noticed the "JVM-Memory" report off Solr's
> home page is fluctuating.  I expect some fluctuation, but it kinda worries
> me when it fluctuates up / down in a range of 4 GB and maybe more.  I.e.:
> at times it is at 5 GB and other times it is at 10 GB (this is while I'm
> running my search tests).  What does such high fluctuation means=3F
>=20
> If it helps, Solr's "JVM-Memory" report states 2.5 GB usage when Solr is
> first started and before I run any search on it.  I'm taking this as my
> base startup memory usage.
>=20
> Steve
>=20
> On Tue, Dec 8, 2015 at 3:17 PM, Erick Erickson <erickerickson@gmail.com>

A follow up question.  I also noticed the "JVM-Memory" report off Solr's
home page is fluctuating.  I expect some fluctuation, but it kinda worries
me when it fluctuates up / down in a range of 4 GB and maybe more.  I.e.:
at times it is at 5 GB and other times it is at 10 GB (this is while I'm
running my search tests).  What does such high fluctuation means=3F

If it helps, Solr's "JVM-Memory" report states 2.5 GB usage when Solr is
first started and before I run any search on it.  I'm taking this as my
base startup memory usage.
> wrote:
>=20
> > You're doing nothing wrong, that particular bit of advice has
> > always needed a bit of explanation.
> >
> > Solr (well, actually Lucene) uses MMapDirectory for much of
> > the index structure which uses the OS memory rather than
> > the JVM heap. See Uwe's excellent:
> >
> > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
> >
> > Plus, the size on disk includes the stored data, which is in the *.fdt
> > files in data/index. Very little of the stored data is kept in the JVM
> > so that's another reason your Java heap may be smaller than
> > your raw index size on disk.
> >
> > The advice about fitting your entire index into memory really has
> > the following caveats (at least).
> > 1> "memory" includes the OS memory available to the process
> > 2> The size of the index on disk is misleading, the *.fdt files
> >      should be subtracted in order to get a truer picture.
> > 3> Both Solr and Lucene create structures in the Java JVM
> >      that are _not_ reflected in the size on disk.
> >
> > <1> and <2> mean the JVM memory necessary is smaller
> > than the size on disk.
> >
> > <3> means the JVM memory will be larger than.
> >
> > So you're doing the right thing, testing and seeing what you
> > _really_ need. I'd pretty much take your test, add some
> > padding and consider it good. You're _not_ doing the
> > really bad thing of using the same query over and over
> > again and hoping <G>.
> >
> > Best,
> > Erick
> >
> >
> > On Tue, Dec 8, 2015 at 11:54 AM, Steven White <swhite4141@gmail.com>
> > wrote:
> > > Hi folks,
> > >
> > > My index size on disk (optimized) is 20 GB (single core, single index).
> > I
> > > have a system with 64 GB of RAM.  I start Solr with 24 GB of RAM.
> > >
> > > I have run load tests (up to 100 concurrent users) for hours where each
> > > user issuing unique searches (the same search is never executed again for
> > > at least 30 minute since it was last executed).  In all tests I run,
> > Solr's
> > > JVM memory never goes over 10 GB (monitoring http://localhost:8983/).
> > >
> > > I read over and over, for optimal performance, Solr should be given
> > enough
> > > RAM to hold the index in memory.  Well, I have done that and some but
> > yet I
> > > don't see Solr using up that whole RAM.  What am I doing wrong=3F  Is my
> > test
> > > at fault=3F  I doubled the test load (number of users) and didn't see much
> > of
> > > a difference with RAM usage but yet my search performance went down
> > (takes
> > > about 40% longer now).  I run my tests again but this time with only 12
> > GB
> > > of RAM given to Solr.  Test result didn't differ much from the 24 GB run
> > > and Solr never used more than 10 GB of RAM.
> > >
> > > Can someone help me understand this=3F  I don't want to give Solr RAM that
> > it
> > > won't use.
> > >
> > > PS: This is simply search tests, there is no update to the index at all.
> > >
> > > Thanks in advanced.
> > >
> > > Steve
> >
>=20