hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-2195) Support cyclic replication
Date Wed, 07 Sep 2011 18:25:15 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099177#comment-13099177
] 

jiraposter@reviews.apache.org commented on HBASE-2195:
------------------------------------------------------



bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java,
line 182
bq.  > <https://reviews.apache.org/r/1730/diff/2/?file=38154#file38154line182>
bq.  >
bq.  >     Yeah, if the clusterids are different, i'd think the edits are different?  That
looks like oversight in original code.
bq.  
bq.  Lars Hofhansl wrote:
bq.      I was not sure here, what does it means for two log keys to be different. I just
noticed that he cluster id was missing.
bq.      We seems to use this specifically for sorting, maybe HLogKey even with different
cluster id need to order the same?

Should HLogKey should sort by sequenceid?


bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java,
line 1001
bq.  > <https://reviews.apache.org/r/1730/diff/2/?file=38153#file38153line1001>
bq.  >
bq.  >     Yeah, can't we pass the cluserid in on the HLog constructor rather than per
append or do we need to allow for different clusterids coming in via append?
bq.  
bq.  Lars Hofhansl wrote:
bq.      Ok.. I'll add a new constructor, then we won't need to call the setter.

Well, sounds like you don't need the constructor.  I was only suggesting it if clusterid was
same for all edits that come into the WAL but later I saw that the edits can come from another
cluster.  If you added clusterid to constructor, what would you do?  Add the passed clusterid
to the WAL IFF the passed Delete or Put were absent a clusterid?


bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java,
line 666
bq.  > <https://reviews.apache.org/r/1730/diff/2/?file=38150#file38150line666>
bq.  >
bq.  >     Hmm.. there is an fb patch that gives Get/Delete, etc. a common  heritage. 
Let me dig it up and commit so you can put this common code there.
bq.  
bq.  Lars Hofhansl wrote:
bq.      I was thinking about this. Then I thought the code duplication is not significant
enough. But now I realize Put and Delete already have a *lot* in common that could be factored
out in addition to this.
bq.      A common superclass maybe "Mutation" would be nice.
bq.      
bq.      I can make that change too, depending on how good the FB code is. Maybe in a separate
jira?
bq.  
bq.  Michael Stack wrote:
bq.      I notice that Put and Delete implement Operation (The Riley slow-query change has
gone in already).  Could you put the dup code up in there?
bq.  
bq.  Lars Hofhansl wrote:
bq.      He, I was thinking about that too. :)
bq.      
bq.      Operation is also shared by Get, Scan, and MultiPut, though.
bq.      Get and Scan do have attributes and MultiPut does not.
bq.      
bq.      The cluster id part really only makes sense for Put and Delete.
bq.      
bq.      It almost seems we want two abstraction:
bq.      1. Attributes. These are common to Put, Delete, Get, and Scan. (Now that I look at
it, I can't believe how often the same code is repeated for those :) ). Maybe have a "WithAttributes"
class for the lack of a better name?
bq.      2. Cluster id, common to Put and Delete.
bq.      
bq.      Put and Delete also have other stuff in common (FamilyMap, writeToWal, the row, the
rowLock, toMap is very similar, etc, etc.)
bq.      
bq.      Is there a jira for the FB change?
bq.      
bq.      So for now... Get this checked in the way it is. Then have additional jira that refactor
Put/Delete and maybe Get/Scan/Multiput?

On 1., I'd say Attributes rather than WithAttributes as the added class name (I think adding
it would be good but can be in a different issue I'd say).

On 2., you talking about a ClusterId or should it be a Replication Interface?  If an Interface,
you can mix it in easily.  But then you don't have a super class to put the commonality into.
 Back to square one (smile).  Or some utility that took the Interface and it did the set and
get of clusterid.

But if Put and Delete have other commonality, that'd be cool figuring a superclass they could
share (Result too?)

But now we are into a different JIRA?  Don't you think?  Yes, lets do the refactor elsewhere.
 The duplicated code don't look so bad after the above exercise teasing stuff out.

The JIRA that added Operation is:

   HBASE-4117  Slow Query Log and Client Operation Fingerprints
               (Riley Patterson)


- Michael


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/#review1780
-----------------------------------------------------------


On 2011-09-06 23:16:48, Lars Hofhansl wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1730/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-09-06 23:16:48)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  For Master <-> Master replication (as well as cycles > 2) a replication sink
needs to keep track who is was the originator of a change and
bq.  (1) do not replicate this data back to the originator and
bq.  (2) pass that information on to the next link in a cycle > 2
bq.  
bq.  In order to do that, HlogKeys read from the WAL are tagged with the Cluster UUID at the
ReplicationSource before the logs are shipped to the Sink. The sink writes the Cluster UUID
into the WAL log (in HlogKey, so only once per edit). If the sink is also source for yet another
sink the ClusterUUID is retained in the HLogKeys read from the local WAL and passed on to
the sink.
bq.  
bq.  
bq.  This addresses bug HBASE-2195.
bq.      https://issues.apache.org/jira/browse/HBASE-2195
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/1730/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  All tests pass.
bq.  New test: TestMasterReplication (which tests Master <-> Master, but no cycles >
2, yet), also passes.
bq.  Manually tested a cycle between 3 clusters.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.



> Support cyclic replication
> --------------------------
>
>                 Key: HBASE-2195
>                 URL: https://issues.apache.org/jira/browse/HBASE-2195
>             Project: HBase
>          Issue Type: Sub-task
>          Components: replication
>            Reporter: Jean-Daniel Cryans
>            Assignee: Lars Hofhansl
>         Attachments: 2195-v5.txt, 2195-v6.txt, 2195.txt
>
>
> We need to support cyclic replication by using the cluster id of each HlogKey and stop
replicating when it goes back to the original cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message