lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Walter Underwood <>
Subject Re: Data Centre recovery/replication, does this seem plausible?
Date Thu, 29 Aug 2013 16:20:56 GMT
Here is a really different approach.

Make the two data centers one Solr Cloud cluster and use a third data center (or EC2 region)
for one additional Zookeeper node. When you lose a DC, Zookeeper still functions.

There would be more traffic between datacenters.


On Aug 29, 2013, at 4:11 AM, Erick Erickson wrote:

> Yeah, reality gets in the way of simple solutions a lot.....
> And making it even more fun you'd really want to only
> bring up one node for each shard in the broken DC and
> let that one be fully synched. Then bring up the replicas
> in a controlled fashion so you didn't saturate the local
> network with replications. And then you'd.....
> But as Shawn says, this is certainly functionality that
> would be waaay cool, there's just been no time to
> make it all work, the main folks who've been working
> in this area all have a mountain of higher-priority
> stuff to get done first....
> There's been talk of making SolrCloud "rack aware" which
> could extend into some kind of work in this area, but
> that's also on the "future" plate. As you're well aware
> it's not a trivial problem!
> Hmmm, what you really want here is the ability to say
> to a recovering cluster "do your initial synch using nodes
> that the ZK ensemble located at XXX know about, then
> switch to your very own ensemble". Something like a
> "remote recovery" option..... Which is _still_ kind of
> tricky, I sure hope you have identical sharding schemes.....
> Erick
> On Wed, Aug 28, 2013 at 1:12 PM, Shawn Heisey <> wrote:
>> On 8/28/2013 10:48 AM, Daniel Collins wrote:
>>> What ideally I would like to do
>>> is at the point that I kick off recovery, divert the indexing feed for the
>>> "broken" into a transaction log on those machines, run the replication and
>>> swap the index in, then replay the transaction log to bring it all up to
>>> date.  That process (conceptually)  is the same as the
>>>**RecoveryStrategy code.
>> I don't think any such mechanism exists currently.  It would be extremely
>> awesome if it did.  If there's not an existing Jira issue, I recommend that
>> you file one.  Being able to set up a multi-datacenter cloud with automatic
>> recovery would be awesome.  Even if it took a long time, having it be fully
>> automated would be exceptionally useful.
>> Yes, if I could divert that feed a that application level, then I can do
>>> what you suggest, but it feels like more work to do that (and build an
>>> external transaction log) whereas the code seems to already be in Solr
>>> itself, I just need to hook it all up (famous last words!) Our indexing
>>> pipeline does a lot of pre-processing work (its not just pulling data from
>>> a database), and since we are only talking about the time taken to do the
>>> replication (should be an hour or less), it feels like we ought to be able
>>> to store that in a Solr transaction log (i.e. the last point in the
>>> indexing pipeline).
>> I think it would have to be a separate transaction log.  One problem with
>> really big regular tlogs is that when Solr gets restarted, the entire
>> transaction log that's currently on the disk gets replayed.  If it were big
>> enough to recover the last several hours to a duplicate cloud, it would
>> take forever to replay on Solr restart.  If the regular tlog were kept
>> small but a second log with the last 24 hours were available, it could
>> replay updates when the second cloud came back up.
>> I do import from a database, so the application-level tracking works
>> really well for me.
>> Thanks,
>> Shawn

Walter Underwood

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message