lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: Solr 4.2 Startup Detects Corrupt Log And is Really Slow to Start
Date Wed, 17 Apr 2013 17:56:20 GMT

On Apr 17, 2013, at 1:42 PM, Shawn Heisey <solr@elyograg.org> wrote:

> On 4/17/2013 10:29 AM, Umesh Prasad wrote:
>> We use DIH and have turned off the Auto commit because we have to sometimes
>> build index from Scratch (clean=true) and we not want to
>> Our master server sees a lot of restarts, sometimes 2-3 times a day. It
>> polls other Data Sources for updates which are quite a few. Master
>> maintains a version of last committed version and can handle uncommitted
>> changes.
>> 
>> Given the frequent restarts, We can't really afford a huge start up at this
>> point.
>>  In the worst case, does Solr allow for disabling transactional log ?
> 
> Unless you are using SolrCloud, you can disable the updateLog. SolrCloud requires it.
> 
> There is one additional caveat - when you disable the updateLog, you have to switch to
MMapDirectoryFactory instead of NRTCachingDirectoryFactory.  The NRT directory implementation
will cache a portion of a commit (including hard commits) into RAM instead of onto disk. 
On the next commit, the previous one is persisted completely to disk.  Without a transaction
log, you can lose data.

I don't think this is true? NRTCachingDirectoryFactory should not cache hard commits and should
be as safe as MMapDirectoryFactory is - neither of which is as safe as using a tran log.

- Mark

> 
> My advice - keep the updateLog on, and use autoCommit with openSearcher=false.  It is
the best way to avoid large transaction logs.  It sounds like you do not want the auto commits
to affect query results, which is a reasonable goal.  You can have that even with autoCommit
- just set openSearcher to false.  Here's an example, no need to stick with the numbers that
I have included:
> 
> <updateHandler class="solr.DirectUpdateHandler2">
>  <autoCommit>
>    <maxDocs>25000</maxDocs>
>    <maxTime>300000</maxTime>
>    <openSearcher>false</openSearcher>
>  </autoCommit>
>  <updateLog />
> </updateHandler>
> 
> Thanks,
> Shawn
> 


Mime
View raw message