lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From S G <sg.online.em...@gmail.com>
Subject Re: Recommended index-size per core
Date Thu, 11 May 2017 22:59:45 GMT
Thanks Toke. Your answer did help me a lot.

But one part about your answer is something that has always been confusing
to be me.

> The JVM heap is not used for caching the index data directly (although it
holds derived data). What you need is free memory on your machine for OS
disk-caching.
> The ideal JVM size is extremely dependent on how you index, query and
adjust the filter-cache (secondarily the other caches, but the filter-cache
tends to be the large one).  A heap of 10GB might very well be fine for
handling your whole 50GB index. If that is on a 64GB machine, the remaining
54GB of RAM (minus the other stuff that is running) ought to ensure a fully
cached index.

How can 50GB index be handled by a 10GB heap?
I am a developer myself and would love to know as many details as possible.
So a long answer would be much appreciated.

Also, on a related note - Are stored fields or doc values part of index
too? Are they stored in JVM or OS-cache? (I would guess latter, but does
that mean JVM is just not required for those or a small percentage?)

Thanks
SG




On Thu, May 11, 2017 at 7:33 AM, Shawn Heisey <apache@elyograg.org> wrote:

> On 5/10/2017 11:52 AM, S G wrote:
> > Is there a recommendation on the size of index that one should host
> > per core?
>
> No, there really isn't.
>
> I can list off a bunch of recommendations, but a whole bunch of things
> that I don't know about your install could make those recommendations
> completely wrong.  An index size that works really well for one person
> might have terrible performance for another.
>
> If you haven't already built it, then there are possibly even things
> that YOU don't know about your install yet that can affect what what you
> need.
>
> https://lucidworks.com/sizing-hardware-in-the-abstract-why-
> we-dont-have-a-definitive-answer/
>
> The only general advice I can give you is this:  You're probably going
> to need more RAM.
>
> Thanks,
> Shawn
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message