lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damien Kamerman <dami...@gmail.com>
Subject Re: Suggester uses lots of 'Page cache' memory
Date Wed, 03 May 2017 00:46:19 GMT
Shalin, yes I think it's a case of the Suggester build hitting the index
all at once. I'm thinking it's hitting all docs, even the ones without
fields relevant to the suggester.

Shawn, I am using ZFS, though I think it's comparable to other setups.
mmap() should still be faster, while the ZFS ARC cache may prefer more
memory that other OS disk caches.

So, it sounds like I enough memory/swap to hold the entire index. When will
the memory be released? On a commit?
https://lucene.apache.org/core/6_5_0/core/org/apache/lucene/store/MMapDirectory.html
talks about a bug on the close().


On 2 May 2017 at 23:07, Shawn Heisey <apache@elyograg.org> wrote:

> On 5/1/2017 10:52 PM, Damien Kamerman wrote:
> > I have a Solr v6.4.2 collection with 12 shards and 2 replicas. Each
> > replica uses about 14GB disk usage. I'm using Solaris 11 and I see the
> > 'Page cache' grow by about 7GB for each suggester replica I build. The
> > suggester index itself is very small. The 'Page cache' memory is freed
> > when the node is stopped. I guess the Suggester component is mmap'ing
> > the entire Lucene index into memory and holding it? Is this expected
> > behavior? Is there a workaround?
>
> I found the following.  The last comment on the answer, the one about
> mmap causing double-buffering with ZFS, is possibly relevant:
>
> https://serverfault.com/a/270604
>
> What filesystem are your indexes on?  If it's ZFS, it could completely
> explain the behavior.  If it's not ZFS, then the only part of it that I
> cannot explain is the fact that the page cache is freed when Solr stops.
>
> If this double-buffering actually means that the memory is allocated
> twice, then I think that ZFS is probably the wrong filesystem to run
> Solr on, unless you have a LOT of spare memory.  You could try changing
> the directory factory to one that doesn't use MMAP, but the suggester
> index factory probably cannot be easily changed.  This is too bad --
> normally MMAP is far more efficient than "standard" filesystem access.
>
> I could be reaching completely wrong conclusions based on the limited
> research I did.
>
> Thanks,
> Shawn
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message