lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rallavagu <rallav...@gmail.com>
Subject Re: Solr hard commit
Date Tue, 27 Oct 2015 23:21:40 GMT
Is it related to this config?

<!-- If true, IndexReaders will be opened/reopened from the IndexWriter
          instead of from the Directory. Hosts in a master/slave setup
          should have this set to false while those in a SolrCloud
          cluster need to be set to true. Default: true
       -->
     <!--
     <nrtMode>true</nrtMode>

this will be true by default.

On 10/27/15 1:24 PM, Erick Erickson wrote:
> Hmm, the tlog. AFAIK, that's because of
> "real time get" (Yonik/Mark/Whoever, please
> correct if wrong).
>
> This is a feature where when fetching a document as
> the result of a search it insures that you get the most
> recent version whether it's been committed or not.....
>
> The tlog isn't relevant for searching however.
>
> Best,
> Erick
>
> On Tue, Oct 27, 2015 at 8:56 AM, Rallavagu <rallavagu@gmail.com> wrote:
>>
>>
>> On 10/27/15 8:43 AM, Erick Erickson wrote:
>>>
>>> bq: So, the updated file(s) on the disk automatically read into memory
>>> as they are Memory mapped?
>>
>> Yes.
>>>
>>>
>>> Not quite sure why you care, curiosity or is there something you're
>>> trying to accomplish?
>>
>> This is out of curiosity. So, I can get better understanding of Solr's
>> memory usage (heap & mmap).
>>
>>>
>>> The contents of the index's segment files are read into virtual memory
>>> by MMapDirectory as needed to satisfy queries. Which is the point of
>>> autowarming BTW.
>>
>>
>> Ok. But, I have noticed that even "tlog" files are memory mapped (output
>> from "lsof") in addition to all other files under "data" directory.
>>
>>>
>>> commit in the following is either hard commit with openSearcher=true
>>> or soft commit.
>>
>>
>> Hard commit is setup with openSearcher=false and softCommit is setup for
>> every 2 min.
>>
>>>
>>> Segments that have been created (closed actually) after the last
>>> commit  are _not_ read at all until the next searcher is opened via
>>> another commit. Nothing is done with these new segments before the new
>>> searcher is opened which you control with your commit strategy.
>>
>>
>> I see. Thanks for the insight.
>>
>>>
>>> Best,
>>> Erick
>>>
>>> On Mon, Oct 26, 2015 at 9:07 PM, Rallavagu <rallavagu@gmail.com> wrote:
>>>>
>>>> Erick, Thanks for clarification. I was under impression that
>>>> MMapDirectory
>>>> is being used for both read/write operations. Now, I see how it is being
>>>> used. Essentially, it only reads from MMapDirectory and writes directly
>>>> to
>>>> disk. So, the updated file(s) on the disk automatically read into memory
>>>> as
>>>> they are Memory mapped?
>>>>
>>>> On 10/26/15 8:43 PM, Erick Erickson wrote:
>>>>>
>>>>>
>>>>> You're really looking at this backwards. The MMapDirectory stuff is
>>>>> for Solr (Lucene, really) _reading_ data from closed segment files.
>>>>>
>>>>> When indexing, there are internal memory structures that are flushed
>>>>> to disk on commit, but these have nothing to do with MMapDirectory.
>>>>>
>>>>> So the question is really moot ;)
>>>>>
>>>>> Best,
>>>>> Erick
>>>>>
>>>>> On Mon, Oct 26, 2015 at 5:47 PM, Rallavagu <rallavagu@gmail.com>
wrote:
>>>>>>
>>>>>>
>>>>>> All,
>>>>>>
>>>>>> Are memory mapped files (mmap) flushed to disk during "hard commit"?
If
>>>>>> yes,
>>>>>> should we disable OS level (Linux for example) memory mapped flush?
>>>>>>
>>>>>> I am referring to following for mmap files for Lucene/Solr
>>>>>>
>>>>>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>>>>>
>>>>>> Linux level flush
>>>>>>
>>>>>>
>>>>>> http://www.cyberciti.biz/faq/linux-stop-flushing-of-mmaped-pages-to-disk/
>>>>>>
>>>>>> Solr's hard and soft commit
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>>>>>
>>>>>> Thanks in advance.

Mime
View raw message