lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Soft commit and new replica types
Date Sun, 09 Dec 2018 16:21:00 GMT
Not quite, 600000. The polling interval is half the commit interval....

This has always bothered me a little bit, I wonder at the utility of a
config param. We already have old-style replication with a
configurable polling interval. Under very heavy indexing loads, it
seems to me that either the tlogs will grow quite large or we'll be
pulling a lot of unnecessary segments across the wire, segments
that'll soon be merged away and the merged segment re-pulled.

Apparently, though, nobody's seen this "in the wild", so it's
theoretical at this point.
On Sun, Dec 9, 2018 at 1:48 AM Vadim Ivanov
<vadim.ivanov@spb.ntk-intourist.ru> wrote:
>
> Thanks, Edward, for clues.
> What bothers me is newSearcher start, warming, cache clear... all that CPU consuming
stuff in my heavy-indexing scenario.
> With NRT I had autoSoftCommit:   <maxTime>300000</maxTime>.
> So I had new Searcher no more than  every 5 min on every replica.
> To have more or less  the same effect with TLOG - PULL collection,
> I suppose, I have to have <autoCommit> :   <maxTime>300000</maxTime>
> (yes, I understand that newSearchers start asynchronously on leader and replicas)
> Am I right?
> --
> Vadim
>
>
> > -----Original Message-----
> > From: Edward Ribeiro [mailto:edward.ribeiro@gmail.com]
> > Sent: Sunday, December 09, 2018 12:42 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Soft commit and new replica types
> >
> > Some insights in the new replica types below:
> >
> > On Sat, December 8, 2018 08:42, Vadim Ivanov <
> > vadim.ivanov@spb.ntk-intourist.ru wrote:
> >
> > >
> > > From Ref guide we have:
> > > " NRT is the only type of replica that supports soft-commits..."
> > > "If TLOG replica does become a leader, it will behave the same as if it
> > > was a NRT type of replica."
> > > Does it mean, that if we do not have NRT replicas in the cluster then
> > > autoSoftCommit section in solconfig.xml Ignored completely (even on TLOG
> > > leader)?
> > >
> >
> > No, not completely. Both TLOG and PULL nodes will periodically poll the
> > leader for changes in index segments' files and download those segments
> > from the leader. If hard commit max time is defined in solrconfig.xml the
> > polling interval of each replica will be half that value. Or else if the
> > soft commit max time is defined then the replicas will use half the soft
> > commit max time as the interval. If neither are defined then the poll
> > interval will be 3 seconds (hard coded). See here:
> > https://github.com/apache/lucene-
> > solr/blob/75b183196798232aa6f2dcaaaab117f309119053/solr/core/src/java/o
> > rg/apache/solr/cloud/ReplicateFromLeader.java#L68-L77
> >
> > If the TLOG is the leader it will index locally and append the doc to
> > transaction log as a NRT node would do as well as it will synchronously
> > replicate the data to other TLOG replicas' transaction logs (PULL nodes
> > don't have transaction logs). But TLOG/PULL replicas doesn't support soft
> > commits nor real time gets, afaik.
> >
> > >
> >
> > > <autoSoftCommit>
> > >   <maxTime>60000</maxTime>
> > > </autoSoftCommit>
> > >
> > > Should we say that in autoCommit section openSearcher is always true in
> > > that case?
> >
> >
> > <autoCommit>
> >   <maxDocs>10000</maxDocs>
> >   <maxTime>30000</maxTime>
> >   <maxSize>512m</maxSize>
> >   <openSearcher>false</openSearcher>
> > </autoCommit>
> >
> > Does it mean that new Searcher always starts on all replicas when hard
> > commit happens on leader?
> >
> >
> > Nope. Or at least, the searcher is not synchronously created. Each non
> > leader replica will periodically fetch the index changes from the leader
> > and open a new searcher to reflect those changes as seen here:
> > https://github.com/apache/lucene-
> > solr/blob/75b183196798232aa6f2dcaaaab117f309119053/solr/core/src/java/o
> > rg/apache/solr/handler/IndexFetcher.java#L653
> > But it's important to note that the potential delay between the leader's
> > hard commit and the other replicas fetching those changes from the leader
> > and opening a new searcher to reflect latest changes.
> >
> > PS: I am still digging these new replica types so I can have misunderstood
> > or missed some aspect of it.
> >
> > Regards,
> > Edward
>

Mime
View raw message