hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Daniel Cryans (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2129) Simple Master/Slave replication
Date Thu, 04 Feb 2010 00:15:28 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829335#action_12829335
] 

Jean-Daniel Cryans commented on HBASE-2129:
-------------------------------------------

Today I set up a replication stream between 2 clusters in 2 different datacenters, each cluster
has 20 nodes and it's replicating on average 50 000 rows per second. I will let it run for
some time to see how it goes.

While setting it up I noticed the following:

 - If a zoo.cfg is present in conf/, it completely overrides any value you may have set directly
on a HBC object so using add_peer.rb doesn't work. As a workaround I moved all the configurations
from that file to hbase-site.xml and deleted the file.

 - When using the add_peer.rb script, it's really important to give the exact hbase.zookeeper.quorum
specified in hbase-site.xml or the master cluster will think he's a slave. It would be nice
to add that notice in the script when it confirms the addresses.

 - When a region server is choosing peers it prints out a message like this: "Considering
19 rs, with ratio 1.9". In this case we should read something more like "Considering 2 rs,
with ratio 0.1". Fix that.

 - When setting up the second cluster, you either have to distcp the hbase folder in HDFS
from the first cluster or recreate all the tables empty. If doing the latter, it can be a
real pain if you have a lot of tables and you don't keep the DDL scripts around so I wrote
a jruby script that reads all the HTableDescriptors from the master cluster and passes them
to a HBaseAdmin configured with the address of the second cluster. I could add it in the patch.

> Simple Master/Slave replication
> -------------------------------
>
>                 Key: HBASE-2129
>                 URL: https://issues.apache.org/jira/browse/HBASE-2129
>             Project: Hadoop HBase
>          Issue Type: Sub-task
>            Reporter: Jean-Daniel Cryans
>            Assignee: Jean-Daniel Cryans
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2129-v2.patch, HBASE-2129.patch
>
>
> We should first build a very simple replication mechanism to validate our assumptions
and get a feel of what replication is in this very distributed context. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message