lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Solr hard commit
Date Wed, 28 Oct 2015 01:56:44 GMT
yep. Although I won't guarantee that the tlog won't be MMapped
even if this is turned off.

On Tue, Oct 27, 2015 at 4:21 PM, Rallavagu <rallavagu@gmail.com> wrote:
> Is it related to this config?
>
> <!-- If true, IndexReaders will be opened/reopened from the IndexWriter
>          instead of from the Directory. Hosts in a master/slave setup
>          should have this set to false while those in a SolrCloud
>          cluster need to be set to true. Default: true
>       -->
>     <!--
>     <nrtMode>true</nrtMode>
>
> this will be true by default.
>
> On 10/27/15 1:24 PM, Erick Erickson wrote:
>>
>> Hmm, the tlog. AFAIK, that's because of
>> "real time get" (Yonik/Mark/Whoever, please
>> correct if wrong).
>>
>> This is a feature where when fetching a document as
>> the result of a search it insures that you get the most
>> recent version whether it's been committed or not.....
>>
>> The tlog isn't relevant for searching however.
>>
>> Best,
>> Erick
>>
>> On Tue, Oct 27, 2015 at 8:56 AM, Rallavagu <rallavagu@gmail.com> wrote:
>>>
>>>
>>>
>>> On 10/27/15 8:43 AM, Erick Erickson wrote:
>>>>
>>>>
>>>> bq: So, the updated file(s) on the disk automatically read into memory
>>>> as they are Memory mapped?
>>>
>>>
>>> Yes.
>>>>
>>>>
>>>>
>>>> Not quite sure why you care, curiosity or is there something you're
>>>> trying to accomplish?
>>>
>>>
>>> This is out of curiosity. So, I can get better understanding of Solr's
>>> memory usage (heap & mmap).
>>>
>>>>
>>>> The contents of the index's segment files are read into virtual memory
>>>> by MMapDirectory as needed to satisfy queries. Which is the point of
>>>> autowarming BTW.
>>>
>>>
>>>
>>> Ok. But, I have noticed that even "tlog" files are memory mapped (output
>>> from "lsof") in addition to all other files under "data" directory.
>>>
>>>>
>>>> commit in the following is either hard commit with openSearcher=true
>>>> or soft commit.
>>>
>>>
>>>
>>> Hard commit is setup with openSearcher=false and softCommit is setup for
>>> every 2 min.
>>>
>>>>
>>>> Segments that have been created (closed actually) after the last
>>>> commit  are _not_ read at all until the next searcher is opened via
>>>> another commit. Nothing is done with these new segments before the new
>>>> searcher is opened which you control with your commit strategy.
>>>
>>>
>>>
>>> I see. Thanks for the insight.
>>>
>>>>
>>>> Best,
>>>> Erick
>>>>
>>>> On Mon, Oct 26, 2015 at 9:07 PM, Rallavagu <rallavagu@gmail.com> wrote:
>>>>>
>>>>>
>>>>> Erick, Thanks for clarification. I was under impression that
>>>>> MMapDirectory
>>>>> is being used for both read/write operations. Now, I see how it is
>>>>> being
>>>>> used. Essentially, it only reads from MMapDirectory and writes directly
>>>>> to
>>>>> disk. So, the updated file(s) on the disk automatically read into
>>>>> memory
>>>>> as
>>>>> they are Memory mapped?
>>>>>
>>>>> On 10/26/15 8:43 PM, Erick Erickson wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> You're really looking at this backwards. The MMapDirectory stuff
is
>>>>>> for Solr (Lucene, really) _reading_ data from closed segment files.
>>>>>>
>>>>>> When indexing, there are internal memory structures that are flushed
>>>>>> to disk on commit, but these have nothing to do with MMapDirectory.
>>>>>>
>>>>>> So the question is really moot ;)
>>>>>>
>>>>>> Best,
>>>>>> Erick
>>>>>>
>>>>>> On Mon, Oct 26, 2015 at 5:47 PM, Rallavagu <rallavagu@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> All,
>>>>>>>
>>>>>>> Are memory mapped files (mmap) flushed to disk during "hard commit"?
>>>>>>> If
>>>>>>> yes,
>>>>>>> should we disable OS level (Linux for example) memory mapped
flush?
>>>>>>>
>>>>>>> I am referring to following for mmap files for Lucene/Solr
>>>>>>>
>>>>>>>
>>>>>>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>>>>>>
>>>>>>> Linux level flush
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://www.cyberciti.biz/faq/linux-stop-flushing-of-mmaped-pages-to-disk/
>>>>>>>
>>>>>>> Solr's hard and soft commit
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>>>>>>
>>>>>>> Thanks in advance.

Mime
View raw message