Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: Toke Eskildsen <te@statsbiblioteket.dk>
To: John Nielsen <jn@mcb.dk>
CC: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
Date: Wed, 17 Apr 2013 15:06:38 +0200
Subject: RE: Solr using a ridiculous amount of memory
Thread-Topic: Solr using a ridiculous amount of memory
Thread-Index: Ac47YEle/ZTYr1g5RlGGrGoYAB8DRgAAQCt5
Message-ID: 
 <2E6A89A648463A4EBF093A9062C16683046532C5FA99@SBMAILBOX1.sb.statsbiblioteket.dk>
References: 
 <CAA72waDHwVq=0qs73szOfVrf9Oum+Kk=OOT3N6fY09pqV_1gFg@mail.gmail.com>
	<1366009259.3998.329.camel@te-prime>
	<CAA72waB15k5akN6hnG3zoukUH29s4+xTAtPn7RYfV0wBTJEtHA@mail.gmail.com>
	<1366025913.3998.387.camel@te-prime>
	<CAA72waCdJUFt3HQJde1fiY=REpvgzD9J=g8r5JOukTO56OYNtQ@mail.gmail.com>
	<CAA72waAireb3at-n3+0jT0m2D+gxHn=A3tJFoXNo0nphFF9RvA@mail.gmail.com>
	<2E6A89A648463A4EBF093A9062C16683046532C5FA96@SBMAILBOX1.sb.statsbiblioteket.dk>,<CAA72waB6yk=vDFmPRaOy-5HLNK9bv8ehC1ybkZEuFUYUg1dS3g@mail.gmail.com>
In-Reply-To: 
 <CAA72waB6yk=vDFmPRaOy-5HLNK9bv8ehC1ybkZEuFUYUg1dS3g@mail.gmail.com>
Accept-Language: en-US, da-DK
Content-Language: en-GB
acceptlanguage: en-US, da-DK
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

John Nielsen [jn@mcb.dk]:
> I never seriously looked at my fieldValueCache. It never seemed to get us=
ed:

> http://screencast.com/t/YtKw7UQfU

That was strange. As you are using a multi-valued field with the new setup,=
 they should appear there. Can you find the facet fields in any of the othe=
r caches?

...I hope you are not calling the facets with facet.method=3Denum? Could yo=
u paste a typical facet-enabled search request?

> Yep. We still do a lot of sorting on dynamic field names, so the field ca=
che
> has a lot of entries. (9.411 entries as we speak. This is considerably lo=
wer
> than before.). You mentioned in an earlier mail that faceting on a field
> shared between all facet queries would bring down the memory needed.
> Does the same thing go for sorting?

More or less. Sorting stores the raw string representations (utf-8) in memo=
ry so the number of unique values has more to say than it does for faceting=
. Just as with faceting, a list of pointers from documents to values (1 val=
ue/document as we are sorting) is maintained, so the overhead is something =
like

#documents*log2(#unique_terms*average_term_length) + #unique_terms*average_=
term_length
(where average_term_length is in bits)

Caveat: This is with the index-wide sorting structure. I am fairly confiden=
t that this is what Solr uses, but I have not looked at it lately so it is =
possible that some memory-saving segment-based trickery has been implemente=
d.

> Does those 9411 entries duplicate data between them?

Sorry, I do not know. SOLR-1111 discusses the problems with the field cache=
 and duplication of data, but I cannot infer if it is has been solved or no=
t. I am not familiar with the stat breakdown of the fieldCache, but it _see=
ms_ to me that there are 2 or 3 entries for each segment for each sort fiel=
d. Guesstimating further, let's say you have 30 segments in your index. Goi=
ng with the guesswork, that would bring the number of sort fields to 9411/3=
/30 ~=3D 100. Looks like you use a custom sort field for each client?

Extrapolating from 1.4M documents and 180 clients, let's say that there are=
 1.4M/180/5 unique terms for each sort-field and that their average length =
is 10. We thus have
1.4M*log2(1500*10*8) + 1500*10*8 bit ~=3D 23MB=20
per sort field or about 4GB for all the 180 fields.

With this few unique values, the doc->value structure is by far the biggest=
, just as with facets. As opposed to the faceting structure, this is fairly=
 close to the actual memory usage. Switching to a single sort field would r=
educe the memory usage from 4GB to about 55MB.

> I do commit a bit more often than i should. I get these in my log file fr=
om
> time to time: PERFORMANCE WARNING: Overlapping onDeckSearchers=3D2

So 1 active searcher and 2 warming searchers. Ignoring that one of the warm=
ing searchers is highly likely to finish well ahead of the other one, that =
means that your heap must hold 3 times the structures for a single searcher=
. With the old heap size of 25GB that left "only" 8GB for a full dataset. S=
ubtract the 4GB for sorting and a similar amount for faceting and you have =
your OOM.

Tweaking your ingest to avoid 3 overlapping searchers will lower your memor=
y requirements by 1/3. Fixing the facet & sorting logic will bring it down =
to laptop size.

> The control panel says that the warm up time of the last searcher is 5574=
. Is that seconds or milliseconds?
> http://screencast.com/t/d9oIbGLCFQwl

milliseconds, I am fairly sure. It is much faster than I anticipated. Are y=
ou warming all the sort- and facet-fields?

> Waiting for a full GC would take a long time.

Until you have fixed the core memory issue, you might consider doing an exp=
licit GC every night to clean up and hope that it does not occur automatica=
lly at daytime (or whenever your clients uses it).

> Unfortunately I don't know of a way to provoke a full GC on command.

VisualVM, which is delivered with the Oracle JDK (look somewhere in the bin=
 folder), is your friend. Just start it on the server and click on the rele=
vant process.

Regards,
Toke Eskildsen=