Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Subject: Re: Spread SolrCloud across two locations
To: solr-user@lucene.apache.org
References: <zarafa.5923fd92.0fdf.38ecc70b07f0306b@mail1.ams.nl.openindex.io>
 <C66B4BAE-1463-462B-852F-57D5FA8F0D66@cominvent.com>
 <CAJuqOK+UcdRbgvRa2agAbxHw+mSN+5QD1GVBef9Z_fX8__Ko+w@mail.gmail.com>
 <18751a00-848f-4bff-a3e5-ab1be7d356fb@elyograg.org>
 <93178F40-D201-4D8C-83A8-234A422C2D61@cominvent.com>
From: Shawn Heisey <apache@elyograg.org>
Message-ID: <f6d07971-b8f1-a02b-977a-f0540b2a0472@elyograg.org>
Date: Wed, 24 May 2017 17:01:48 -0600
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.8.0
MIME-Version: 1.0
In-Reply-To: <93178F40-D201-4D8C-83A8-234A422C2D61@cominvent.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
archived-at: Wed, 24 May 2017 23:02:17 -0000

On 5/24/2017 4:14 PM, Jan Høydahl wrote:
> Sure, ZK does by design not support a two-node/two-location setup. But still, users may want/need to deploy that,
> and my question was if there are smart ways to make such a setup as little painful as possible in case of failure.
>
> Take the example of DC1: 3xZK and DC2: 2xZK again. And then DC1 goes BOOM.
> Without an active action DC2 would be read-only
> What if then the Ops personnel in DC2 could, with a single script/command, instruct DC2 to resume “master” role:
> - Add a 3rd DC2 ZK to the two existing, reconfigure and let them sync up.
> - Rolling restart of Solr nodes with new ZK_HOST string
> Of course, they would also then need to make sure that DC1 does not boot up again before compatible change has been done there too.

When ZK 3.5 comes out and SolrCloud is updated to use it, I think that
it might be possible to remove the dc1 servers from the ensemble and add
another server in dc2 to re-form a new quorum, without restarting
anything.  It could be quite some time before a stable 3.5 is available,
based on past release history.  They don't release anywhere near as
often as Lucene/Solr does.

With the current ZK version, I think your procedure would work, but I
definitely wouldn't call it painless.  Indexing would be unavailable
when dc1 goes down, and everything could be down while the restarts are
happening.

Whether ZK 3.5 is there or not, there is potential unknown behavior when
dc1 comes back online, unless you can have dc1 personnel shut the
servers down, or block communication between your servers in dc1 and dc2.

Overall, having one or two ZK servers in each main DC and a tiebreaker
ZK on a low-cost server in a third DC seems like a better option. 
There's no intervention required when a DC goes down, or when it comes
back up.

Thanks,
Shawn