Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
MIME-Version: 1.0
In-Reply-To: <BANLkTi=91snyKRuG-cr7d5ms7RFaGcRvDw@mail.gmail.com>
References: <BANLkTi=91snyKRuG-cr7d5ms7RFaGcRvDw@mail.gmail.com>
Date: Thu, 12 May 2011 21:17:52 -0700
Message-ID: <BANLkTikFcgLUTO7nOYeQ+mAVxXA+GOkiTw@mail.gmail.com>
Subject: Re: Gracefully changing IP addresses of quorum members?
From: Benjamin Reed <breed@apache.org>
To: user@zookeeper.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

sorry i realized i only answered half of your question. you can move
the cluster. you do the rolling configuration change that ted dunning
has outlined in the past.

ben

On Wed, May 11, 2011 at 12:53 PM, tsuna <tsunanet@gmail.com> wrote:
> Hi all,
> I have a 5-node quorum with 1 node per rack. =A0It's unfortunate but it
> so happens that I need to free up these 5 racks and move all their
> machines somewhere else. =A0This requires me to change the IP addresses
> of all the machines in these 5 racks as they land in other racks.
>
> All our applications are configured with a quorum specification of
> "zookeeper". =A0This ends up being resolved as
> zookeeper.datacenter.internaldomain.com, which returns 5 A records.
> All our Java applications are also started with
> -Dsun.net.inetaddr.ttl=3D600, just in case that matters, because the JDK
> by default has the wonderful idea to cache all DNS names forever
> regardless of the TTL. =A0So with this flag the JDK will cache DNS
> entries for 10 minutes maximum.
>
> I reviewed the FAQ (http://wiki.apache.org/hadoop/ZooKeeper/FAQ =96 BTW
> there's a broken image in answer to the 1st question) and the admin
> guide (http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html) and
> searched on the intertubes a bit, but I couldn't come up with an
> answer.
>
> Can I move the ZK machines one by one and re-IP them and hope they'll
> rejoin the quorum gracefully? =A0If I allow like 20-30 minutes between
> each machine move, we should always have a quorum, provided that the
> machines can re-join the quorum with a different IP. =A0Is that the
> case? =A0Has anyone tested this?
>
> If that doesn't work, can you think of anything I can do to gracefully
> move the quorum?
>
> Thanks.
>
> --
> Benoit "tsuna" Sigoure
> Software Engineer @ www.StumbleUpon.com
>