lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Tsadikov <mich...@myheritage.com>
Subject Re: upgrading 1hr autoCommit behavior
Date Tue, 11 Jun 2013 22:07:55 GMT
Thanks for the quick reply, Shawn.

I am less worried about long transaction-log replays after crashes because
in normal life (long uptimes, rare orderly restarts) this should never
happen, and even if it does, uses will not be affected.

I am more worried about potential increase in heap usage due to
transaction-logs, and possible affect on indexer stability, and am
considering increasing the JVM heap - do you think this worry is realistic?

Thanks,
Michael


On Wed, Jun 12, 2013 at 12:58 AM, Shawn Heisey <elyograg@elyograg.org>wrote:

> On 6/11/2013 3:45 PM, Michael Tsadikov wrote:
>
>> We're upgrading our Solr 3.1 distributed masters/slaves setup to 4.3.
>>
>> In 3.1 we used autoCommit every hour on masters, each commit is replicated
>> to slaves, and all searches are done on slaves. 1hr visibility is ok - we
>> don't need NRT.
>>
>> In 4.3 we enable transaction logs, for durability in case of crashes.
>>
>> However, we are wondering how to best setup autoCommit/autoSoftCommit on
>> masters to preserve the old behavior. It seems that setting autoCommit to
>> 1hr (openSearcher=true) without any autoSoftCommit preserves our previous
>> setup - is this correct? Wil the transaction log make masters use much
>> more
>> heap due to 1hr periods between commits? This can be a problem for us
>> because we put many master cores on one solr JVM
>>
>
> If you want to completely preserve your previous setup, then you've got it
> correct.  Depending on how much you index over the course of that hour, you
> might want to go a different way.
>
> If you are doing hard commits once an hour, you'll end up with at least 2
> hours of indexing in your transaction log.  If a few hours of indexing
> represents a lot of data, then it can take a really long time to replay
> those transactions when Solr restarts.  If it's not a lot of data, then you
> shouldn't have any trouble.
>
> http://wiki.apache.org/solr/**SolrPerformanceProblems#Slow_**startup<http://wiki.apache.org/solr/SolrPerformanceProblems#Slow_startup>
>
> If a few hours of indexing data *IS* a lot, then autoCommit every five
> minutes with openSearcher=false and autoSoftCommit once an hour would
> preserve your current visibility characteristics without giving you really
> huge transaction logs.
>
> Thanks,
> Shawn
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message