lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rallavagu <rallav...@gmail.com>
Subject Re: Solr hard commit
Date Tue, 27 Oct 2015 15:56:12 GMT


On 10/27/15 8:43 AM, Erick Erickson wrote:
> bq: So, the updated file(s) on the disk automatically read into memory
> as they are Memory mapped?
Yes.
>
> Not quite sure why you care, curiosity or is there something you're
> trying to accomplish?
This is out of curiosity. So, I can get better understanding of Solr's 
memory usage (heap & mmap).

>
> The contents of the index's segment files are read into virtual memory
> by MMapDirectory as needed to satisfy queries. Which is the point of
> autowarming BTW.

Ok. But, I have noticed that even "tlog" files are memory mapped (output 
from "lsof") in addition to all other files under "data" directory.

>
> commit in the following is either hard commit with openSearcher=true
> or soft commit.

Hard commit is setup with openSearcher=false and softCommit is setup for 
every 2 min.

>
> Segments that have been created (closed actually) after the last
> commit  are _not_ read at all until the next searcher is opened via
> another commit. Nothing is done with these new segments before the new
> searcher is opened which you control with your commit strategy.

I see. Thanks for the insight.

>
> Best,
> Erick
>
> On Mon, Oct 26, 2015 at 9:07 PM, Rallavagu <rallavagu@gmail.com> wrote:
>> Erick, Thanks for clarification. I was under impression that MMapDirectory
>> is being used for both read/write operations. Now, I see how it is being
>> used. Essentially, it only reads from MMapDirectory and writes directly to
>> disk. So, the updated file(s) on the disk automatically read into memory as
>> they are Memory mapped?
>>
>> On 10/26/15 8:43 PM, Erick Erickson wrote:
>>>
>>> You're really looking at this backwards. The MMapDirectory stuff is
>>> for Solr (Lucene, really) _reading_ data from closed segment files.
>>>
>>> When indexing, there are internal memory structures that are flushed
>>> to disk on commit, but these have nothing to do with MMapDirectory.
>>>
>>> So the question is really moot ;)
>>>
>>> Best,
>>> Erick
>>>
>>> On Mon, Oct 26, 2015 at 5:47 PM, Rallavagu <rallavagu@gmail.com> wrote:
>>>>
>>>> All,
>>>>
>>>> Are memory mapped files (mmap) flushed to disk during "hard commit"? If
>>>> yes,
>>>> should we disable OS level (Linux for example) memory mapped flush?
>>>>
>>>> I am referring to following for mmap files for Lucene/Solr
>>>>
>>>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>>>
>>>> Linux level flush
>>>>
>>>> http://www.cyberciti.biz/faq/linux-stop-flushing-of-mmaped-pages-to-disk/
>>>>
>>>> Solr's hard and soft commit
>>>>
>>>>
>>>> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>>>
>>>> Thanks in advance.

Mime
View raw message