Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates
 209.85.161.50 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=sVNXcZvZmcdc354G2JhzDUK+HTsCNEK7orQhqgvjyjtBeg4fuXkyF/bQkTpZ6OP6Zc
         r8wROiDCfoPZ/VknQS6WOAvh5OZA6CeQHT6hzu3K5XVjTomT/zQmnRcDZC6t2e9ojAhf
         oFK2dZt2PCxNO6vNUmkHES3G+I14TMNNiv7GQ=
MIME-Version: 1.0
Sender: jdcryans@gmail.com
In-Reply-To: <AANLkTik32YWgk_dcEfYe6j3uvH2a9rsqSyML4uCQwREe@mail.gmail.com>
References: <AANLkTik32YWgk_dcEfYe6j3uvH2a9rsqSyML4uCQwREe@mail.gmail.com>
Date: Mon, 13 Dec 2010 11:28:32 -0800
Message-ID: <AANLkTikhkZMoqsM7zfAFtjTSS6n-FWtHB22P==dpezNJ@mail.gmail.com>
Subject: Re: HBase Replication problems
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Nathaniel,

Thanks for trying out replication, let's make it work for you.

So on the master-side there's 2 lines that are important to make sure
that replication works, first it has to say:

Replicating x

Where x is the number of edits it's going to ship, and then

Replicated in total: y

Where y is the total number it replicated. Seeing the second line
means that replication was successful, at least from the master point
of view.

On the slave, one node should have:

Total replicated: z

And that z is the number of edits that that region server applied on
it's cluster. It could be on any region server, since the sink for
replication is chose at random.

Do you see those? Any exceptions around those logs apart from EOFs?

Thx,

J-D

On Mon, Dec 13, 2010 at 10:52 AM, Nathaniel Cook
<nathanielc@qualtrics.com> wrote:
> Hi,
>
> I am trying to setup replication for my HBase clusters. I have two
> small clusters for testing each with 4 machines. The setup for the two
> clusters is identical. Each machine runs a DataNode, and
> HRegionServer. Three of the machines run a ZK peer and one machine
> runs the HMaster and NameNode. The cluster master machines have
> hostnames (ds1,ds2 ...) and the slave cluster is (bk1, bk2 ...). I set
> the replication =A0scope to 1 for my test table column families and set
> the hbase.replication property to true for both clusters. Next I ran
> the add_peer.rb script with the following command on the ds1 machine:
>
> hbase org.jruby.Main /usr/lib/hbase/bin/replication/add_peer.rb
> ds1:2181:/hbase bk1:2181:/hbase
>
> After the script finishes ZK for the master cluster has the
> replication znode and children of peers, master, and state. The slave
> ZK didn't have a replication znode. I fixed that problem by rerunning
> the script on the bk1 machine and commenting out the code to write to
> the master ZK. Now the slave ZK has the /hbase/replication/master
> znode with data (ds1:2181:/hbase). Everthing looked to be configured
> correctly. I restarted the clusters. The logs of the master
> regionservers stated:
>
> This cluster (ds1:2181:/hbase) is a master for replication, compared
> with (ds1:2181:/hbase)
>
> The logs on the slave cluster stated:
>
> This cluster (bk1:2181:/hbase) is a slave for replication, compared
> with (ds1:2181:/hbase)
>
> Using the hbase shell I put a row into the test table.
>
> The regionserver for that table had a log statement like:
>
> Going to report log #192.168.1.166%3A60020.1291757445179 for position
> 15828 in hdfs://ds1:9000/hbase/.logs/ds1.internal,60020,1291757445059/192=
.168.1.166
> <http://192.168.1.166/>%3A60020.1291757445179
>
> (192.168.1.166 is ds1)
>
> I wait and even after several minutes the row still does not appear in
> the slave cluster table.
>
> Any help with what the problem might be is greatly appreciated.
>
> Both clusters are using a CDH3b3. The HBase version is exactly
> 0.89.20100924+28.
>
> -Nathaniel Cook
>