Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: message received from 54.76.25.247 which
 is an MX secondary for user@hbase.apache.org)
MIME-Version: 1.0
From: Chandrashekhar Kotekar <shekhar.kotekar@gmail.com>
Date: Tue, 28 Apr 2015 16:34:32 +0530
Message-ID: 
 <CAJwfZdornLzBX7xCGKHc8tk-P7KdeQrfVENDHZJ4SySPBX93Bw@mail.gmail.com>
Subject: Cluster replication is not replicating any data
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a113cd5e8f2dd590514c6d4a3

--001a113cd5e8f2dd590514c6d4a3
Content-Type: text/plain; charset=UTF-8

Hi,

I have setup cluster replication between 2 clusters but data is not getting
copied. Can anyone please help me in cluster replication?

I would like to know if I add only 1 row in one of the tables then will
that row get replicated in other cluster or HBase waits for some time or
HBase waits before certain amount of data is added/deleted/edited (like
64MB)?

Clusters are based on CDH 5.3.1 which have 0.98.6 version of HBase. I have
named slave nodes of both the clusters as slave1 and slave2 and master
server is named as hbase-master (This setting is in /etc/hosts file).

I took following steps to setup replication:
1) Set value of 'hbase.replication' property in 'hbase-site.xml' file to
true in both the clusters

2) Restarted HBase master and all region servers of both the clusters.
(Restarting needs to be done in any specifi order?? like master first then
region server or vice versa??)

3) Disabled, Altered all the tables in both the clusters by adding "{NAME
=> 'CF', REPLICATION_SCOPE => 1}" in alter statement, and then enabled
tables one by one. (CF is actual column family name)

4) Added peer using "add_peer '1', '10.0.21.249:2181:/hbase' " in primary
cluster.

5) added peer using "add_peer 'nc' '10.0.21.111:/hbase '" in target cluster.

6) Primary cluster already has 4 rows. Added one more row having one column
for COMPANY_TABLE into primary cluster. So expecting either total 5 rows in
target cluster or at least newly added row in target cluster. Waited few
minutes for row to get replicated into other cluster. Executed scan
'table_name' command in target cluster but didnt get that row from primary
cluster.

7) Executed following verify replication command in primary cluster. This
command runs m-r job and shows 4 good rows but in target cluster no row
present for COMPANY_TABLE table.

hbase org.apache.hadoop.hbase.mapreduce.replication.VerifyReplication
--families=CF nc COMPANY_TABLE


There's no error in region server logs. Sample logs from primary cluster's
region server is as as below:

15/04/28 10:48:16 INFO regionserver.HRegionServer: Adding moved region
record: 1a685b27708dfea86bb2d8a9ca1bceb5 to
ip-10-0-21-90.ec2.internal,60020,1430217861644:60020 as of 4873480
15/04/28 10:51:06 INFO regionserver.Replication: Normal source for cluster
nc: Total replicated edits: 0, currently replicating from:
hdfs://StagingCluster/hbase/WALs/slave2,60020,1430201462672/slave2%2C60020%2C1430201462672.1430215999727
at position: 83

15/04/28 10:56:06 INFO regionserver.Replication: Normal source for cluster
nc: Total replicated edits: 0, currently replicating from:
hdfs://StagingCluster/hbase/WALs/slave2,60020,1430201462672/slave2%2C60020%2C1430201462672.1430215999727
at position: 83

15/04/28 11:01:06 INFO regionserver.Replication: Normal source for cluster
nc: Total replicated edits: 0, currently replicating from:
hdfs://StagingCluster/hbase/WALs/slave2,60020,1430201462672/slave2%2C60020%2C1430201462672.1430215999727
at position: 83

Regards,
Chandrash3khar Kotekar
Mobile - +91 8600011455

--001a113cd5e8f2dd590514c6d4a3--