lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Umesh Prasad <umesh.i...@gmail.com>
Subject Re: Solr 4.2 Startup Detects Corrupt Log And is Really Slow to Start
Date Tue, 23 Apr 2013 03:31:28 GMT
Sorry for late reply. I was trying to change our indexing pipeline and do
explicit intermediate commits for each core. That turned out to be a bit
more work that I have time for.

    So, I do want to explore hard commits.  I tried
<solr-host>:<port>/solr/<core>/*update?commit=true* . But there is no
impact on Txn Log size, so I feel, it must be getting ignored.

    So can someone tell me, how to do the Hard Commits ?

@Shawn : openSearcher = false is not an option. On Each commit, index will
be replicated to Slaves which will have a searcher on it immediately and
can intermediate state. The longer term and better solution is changing
indexing pipeline and doing explicit commits, but I can't implement that
right now.




On 18 Apr 2013 00:35, "Shawn Heisey" <solr@elyograg.org> wrote:

> On 4/17/2013 11:56 AM, Mark Miller wrote:
>
>> There is one additional caveat - when you disable the updateLog, you have
>>> to switch to MMapDirectoryFactory instead of NRTCachingDirectoryFactory.
>>>  The NRT directory implementation will cache a portion of a commit
>>> (including hard commits) into RAM instead of onto disk.  On the next
>>> commit, the previous one is persisted completely to disk.  Without a
>>> transaction log, you can lose data.
>>>
>>
>> I don't think this is true? NRTCachingDirectoryFactory should not cache
>> hard commits and should be as safe as MMapDirectoryFactory is - neither of
>> which is as safe as using a tran log.
>>
>
> This is based on observations of what happens with my segment files when I
> do a full-import, using autoCommit with openSearcher disabled.  I see that
> each autoCommit results in a full segment being written, the part of
> another segment.  On the next autoCommit, the rest of the files for the
> last segment are written, another full segment is written, I get another
> partial segment.  I asked about this on the list some time ago, and what I
> just told Umesh is a rehash of what I understood from Yonik's response.
>
> If I'm wrong, I hope someone who knows for sure can correct me.
>
> Thanks,
> Shawn
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message