Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
MIME-Version: 1.0
In-Reply-To: <18d72d99-3eff-b46a-c618-178716e0959c@elyograg.org>
References: <CAC9-YrUN9=TqkPQYwP-NkBkLQZmDryMCyQJiu_9JJbhHxcD_hg@mail.gmail.com>
 <c107e7cb-96f4-ac24-7f1d-8b8b470ef855@elyograg.org> <CAC9-YrWHM2Cwxtxu7k_8er=tNkwmMYmz11QVp_CqYUSQyRpbKg@mail.gmail.com>
 <18d72d99-3eff-b46a-c618-178716e0959c@elyograg.org>
From: Damien Kamerman <damienk@gmail.com>
Date: Wed, 3 May 2017 12:47:01 +1000
Message-ID: <CAC9-YrUhBoHgaq=frZO9qfbnEUi_KJnkxmM3ydTroKTtkF6Wdw@mail.gmail.com>
Subject: Re: Suggester uses lots of 'Page cache' memory
To: solr-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=001a113ca9e2b97d1b054e95ab13
archived-at: Wed, 03 May 2017 02:47:08 -0000

--001a113ca9e2b97d1b054e95ab13
Content-Type: text/plain; charset=UTF-8

Thanks Shawn, I'll have to look closer into this.

On 3 May 2017 at 12:10, Shawn Heisey <apache@elyograg.org> wrote:

> On 5/2/2017 6:46 PM, Damien Kamerman wrote:
> > Shalin, yes I think it's a case of the Suggester build hitting the index
> > all at once. I'm thinking it's hitting all docs, even the ones without
> > fields relevant to the suggester.
> >
> > Shawn, I am using ZFS, though I think it's comparable to other setups.
> > mmap() should still be faster, while the ZFS ARC cache may prefer more
> > memory that other OS disk caches.
> >
> > So, it sounds like I enough memory/swap to hold the entire index. When
> will
> > the memory be released? On a commit?
> > https://lucene.apache.org/core/6_5_0/core/org/apache/
> lucene/store/MMapDirectory.html
> > talks about a bug on the close().
>
> What I'm going to describe below is how things *normally* work on most
> operating systems (think Linux or Windows) with most filesystems.  If
> ZFS is different, and it sounds like it might be, then that's something
> for you to discuss with Oracle.
>
> Normally, MMap doesn't *allocate* any memory -- so there's nothing to
> release later.  It asks the operating system to map the file's contents
> to a section of virtual memory, and then the program accesses that
> memory block directly.
>
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> A typical OS takes care of translating accesses to MMap virtual memory
> into disk accesses, and uses available system memory to cache the data
> that's read so a subsequent access of the same data is super fast.
>
> On most operating systems, memory in the disk cache is always available
> to programs that request it for an allocation.
>
> ZFS uses a completely separate piece of memory for caching -- the ARC
> cache.  I do not know if the OS is able to release memory from that
> cache when a program requests it.  My experience with ZFS on Linux  (not
> with Solr) suggests that the ARC cache holds onto memory a lot tighter
> than the standard OS disk cache.  ZFS on Solaris might be a different
> animal, though.
>
> I'm finding conflicting information regarding MMap problems on ZFS.
> Some sources say that memory usage is doubled (data in both the standard
> page cache and the arc cache), some say that this is not a general
> problem.  This is probably a question for Oracle to answer.
>
> You don't want to count swap space when looking at how much memory you
> have.  Swap performance is REALLY bad.
>
> Thanks,
> Shawn
>
>

--001a113ca9e2b97d1b054e95ab13--