Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (nike.apache.org: domain of ted.dunning@gmail.com
 designates 209.85.220.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=MH56UTd1tlYAH49oGKYR+fRvrq/sVWdz1NvQQXLQEjxyYAgvxN7E+05Zdh4oHqsesC
         cDI4igoOAZdNNZDcz7iB0JV/aLq36e1EvCuYZbfWiJpttydIzDHudauLgKgEnFhjzFjp
         CNuxsYaV16czEFf7uBCk2f7//8glKsceIV+To=
MIME-Version: 1.0
In-Reply-To: <BANLkTimeEh0=q+PR0h_PrqTy8Vy-TL+ppg@mail.gmail.com>
References: <BANLkTimeEh0=q+PR0h_PrqTy8Vy-TL+ppg@mail.gmail.com>
From: Ted Dunning <ted.dunning@gmail.com>
Date: Mon, 11 Apr 2011 08:14:17 -0700
Message-ID: <BANLkTims8bWRP6oVfwjo0C_ri-JGBN0sqg@mail.gmail.com>
Subject: Re: Repair cluster on EC2
To: user@zookeeper.apache.org
Cc: Andrei Savu <savu.andrei@gmail.com>, zookeeper-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec548a887b46be304a0a6082f

--bcaec548a887b46be304a0a6082f
Content-Type: text/plain; charset=UTF-8

On Mon, Apr 11, 2011 at 4:43 AM, Andrei Savu <savu.andrei@gmail.com> wrote:

> Is it possible to repair a ZooKeeper cluster on EC2 by using the
> following algorithm with no downtime and data loss?
>

Yes.  Been there.  Done that.  Works like a champ.

1. start a cluster with >3 nodes
> 2. if one node fails start a new machine and record the new IP
> 3. rebuild the configuration file by replacing the IP of the node that
> failed with the IP attached to the new machine
> 4. do a rolling restart and replace all configuration files
>
> Am I missing something? Could this process be executed by a script?
>

Sounds right and you should be able to do it with a script.

Use caution, of course.


> I'm also thinking about extending the client library in order to make
> it EC2 aware (it should be able to automatically discover ZK nodes).
>

This way lies danger!

The problem is that is cluster membership becomes very flexible then you run
the risk of diluting the guarantees that ZK provides based on the quorum
requirements.

--bcaec548a887b46be304a0a6082f--