hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Demai Ni (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8663) a HBase Shell command to list the tables replicated (from or to) current cluster
Date Tue, 30 Jul 2013 17:39:51 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13724140#comment-13724140
] 

Demai Ni commented on HBASE-8663:
---------------------------------

[~ctrezzo]  about your first point on multiple masters. I am thinking about two approaches,
both involve leveraging existing replication_scope(and prerequisites HBASE-8771 , which will
ensure only 0 and 1 be used as replication_scope) 
A simple approach is: set replication scope to -1 if the table/columnfam is on slave-side.
In that case the shell command can be used as below:
{code}
hbase(main):001:0> list_replicated_tables
 TABLE      COLUMNFAMILY       CLUSTER
 scores      course            MASTER: global
 t3_dn       cf1               MASTER: global
 usertable   family            MASTER: global
 t1_dn       cf1               SLAVE
 t2_dn       cf2               SLAVE
{code}


A complex/tricky approach is to change the replication_scope dramatically, and or just invent
a new attribute 'replication_peers', an array of ID. We can use positive ID for target-cluster,
and negative ID for source-cluster, for example 
{code}
hbase(main):004:0> list_peers
 PEER_ID CLUSTER_KEY STATE
 1 Slave_A.hbase.com:2181:/hbase ENABLED
 2 Slave_B.hbase.com:2181:/hbase ENABLED
 3 Slave_Master_C.hbase.com:2181:/hbase ENABLED
-1 Master_A.hbase.com:2181:/hbase ENABLED
-2 Master_B.hbase.com:2181:/hbase ENABLED
-3 Slave_Master_C.hbase.com:2181:/hbase ENABLED
>describe table
't1_dn', {NAME => 'cf1', REPLICATION_PEERS => '1,2,3', ..}
't2_dn', {NAME => 'cf1', REPLICATION_PEERS => '-1,-2',..}
't3_dn', {NAME => 'cf1', REPLICATION_PEERS => '3,-3',..}

t1_dn#cf1 is replicated from this cluster, and its slave clusters are Slave_A,Slave_B and
Slave_Master_C
t2_dn#cf1 is replicated to this cluster, and its master clusters are Master_A and Master_B
t3_dn#cf1 is setup as Master_Slave replication, with Slave_Master_C.hbase.com(while don't
have to be the same cluster) 
{code}   

                
> a HBase Shell command to list the tables replicated (from or to) current cluster
> --------------------------------------------------------------------------------
>
>                 Key: HBASE-8663
>                 URL: https://issues.apache.org/jira/browse/HBASE-8663
>             Project: HBase
>          Issue Type: New Feature
>          Components: Replication, shell
>         Environment: clusters setup as Master and Slave for replication of tables 
>            Reporter: Demai Ni
>            Assignee: Demai Ni
>            Priority: Critical
>         Attachments: HBASE-8663.PATCH, HBASE-8663-trunk-v0.patch, HBASE-8663-v2.PATCH
>
>
> This jira is to provide a hbase shell command which can give user can overview of the
tables/columnfamilies currently being replicated. The information will help system administrator
for design and planning, and also help application programmer to know which tables/columns
should be watchout(for example, not to modify a replicated columnfamily on the slave cluster)
> Currently there is no easy way to tell which table(s)/columnfamily(ies) 
> replicated from or to a particular cluster. 
> 	
> On Master Cluster, an indirect method can be used by combining two steps: 1) $describe
'usertable'  and 2)  $list_peers to map the REPLICATION_SCOPE to target(aka slave) cluster
  
> 	
> On slave cluster, this is no existing API/methods to list all the tables replicated to
this cluster.    
> Here is an example, and prototype for Master cluster
> {code: title=hbase shell command:list_replicated_tables |borderStyle=solid}
> hbase(main):001:0> list_replicated_tables
>  TABLE      COLUMNFAMILY       TARGET_CLUSTER
>  scores      course            hdtest017.svl.ibm.com:2181:/hbase
>  t3_dn       cf1               hdtest017.svl.ibm.com:2181:/hbase
>  usertable   family            hdtest017.svl.ibm.com:2181:/hbase
> 3 row(s) in 0.3380 seconds
> {code}
> {code: title=method to return all columnfamilies replicated from this cluster |borderStyle=solid}
> /** 
>  * ReplicationAdmin.listRepllicated
>    * @return List of the replicated columnfamilies of this cluster for display. 
>    * @throws IOException 
> */
> public List<String[]> listReplicated() throws IOException {
>     List<String[]> replicatedColFams = new ArrayList<String[]>();  
>     
>     HTableDescriptor[] tables;
>     
>     tables= this.connection.listTables();
>     
>     Map<String, String> peers = listPeers();
>     
>     for (HTableDescriptor table:tables) {
>       HColumnDescriptor[] columns = table.getColumnFamilies();
>       String tableName = table.getNameAsString();
>       for (HColumnDescriptor column: columns) {
>         int scope = column.getScope();
>                 
>         if (scope!=0) {
>           String[] replicatedEntry = new String[3]; 
>           replicatedEntry[0] = tableName;
>           replicatedEntry[1] = column.getNameAsString();
>           replicatedEntry[2] = peers.get(Integer.toString(scope));
>           replicatedColFams.add(replicatedEntry);
>         }
>       }
>     }
>     
>     return replicatedColFams;
>   }
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message