hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-2195) Support cyclic replication
Date Wed, 07 Sep 2011 15:44:11 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099040#comment-13099040
] 

jiraposter@reviews.apache.org commented on HBASE-2195:
------------------------------------------------------



bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java,
line 232
bq.  > <https://reviews.apache.org/r/1730/diff/2/?file=38154#file38154line232>
bq.  >
bq.  >     We could have read garbage that happened to be <0?  But yeah, this is going
to go out with new major version and we'll just pronounce that it will not be able to read
old WALs (this we've said in the past)
bq.  >     
bq.  >     Why not VersionedWritable since it does this version stuff for you?
bq.  
bq.  Lars Hofhansl wrote:
bq.      This was a good idea from Ted.
bq.      The first entry in HLogKey is the encodedRegionName which is written by Bytes.writeByteArray.
bq.      The first byte (or two bytes) encode the length of the array.
bq.      
bq.      So we read that first. If the length is < 0 we know it couldn't have been a valid
HLogKey, and hence it must be a new one and this is the now the version.

That sounds like a good idea to me too.  Does it need a comment so you and Ted don' t have
to answer same question everytime some else looks at this bit of code? 

Good stuff Lars.


bq.  On 2011-09-07 06:01:09, Michael Stack wrote:
bq.  > http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java,
line 666
bq.  > <https://reviews.apache.org/r/1730/diff/2/?file=38150#file38150line666>
bq.  >
bq.  >     Hmm.. there is an fb patch that gives Get/Delete, etc. a common  heritage. 
Let me dig it up and commit so you can put this common code there.
bq.  
bq.  Lars Hofhansl wrote:
bq.      I was thinking about this. Then I thought the code duplication is not significant
enough. But now I realize Put and Delete already have a *lot* in common that could be factored
out in addition to this.
bq.      A common superclass maybe "Mutation" would be nice.
bq.      
bq.      I can make that change too, depending on how good the FB code is. Maybe in a separate
jira?

I notice that Put and Delete implement Operation (The Riley slow-query change has gone in
already).  Could you put the dup code up in there?


- Michael


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1730/#review1780
-----------------------------------------------------------


On 2011-09-06 23:16:48, Lars Hofhansl wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/1730/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-09-06 23:16:48)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu, Michael Stack, and Jean-Daniel Cryans.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  For Master <-> Master replication (as well as cycles > 2) a replication sink
needs to keep track who is was the originator of a change and
bq.  (1) do not replicate this data back to the originator and
bq.  (2) pass that information on to the next link in a cycle > 2
bq.  
bq.  In order to do that, HlogKeys read from the WAL are tagged with the Cluster UUID at the
ReplicationSource before the logs are shipped to the Sink. The sink writes the Cluster UUID
into the WAL log (in HlogKey, so only once per edit). If the sink is also source for yet another
sink the ClusterUUID is retained in the HLogKeys read from the local WAL and passed on to
the sink.
bq.  
bq.  
bq.  This addresses bug HBASE-2195.
bq.      https://issues.apache.org/jira/browse/HBASE-2195
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Put.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
1165864 
bq.    http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/1730/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  All tests pass.
bq.  New test: TestMasterReplication (which tests Master <-> Master, but no cycles >
2, yet), also passes.
bq.  Manually tested a cycle between 3 clusters.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Lars
bq.  
bq.



> Support cyclic replication
> --------------------------
>
>                 Key: HBASE-2195
>                 URL: https://issues.apache.org/jira/browse/HBASE-2195
>             Project: HBase
>          Issue Type: Sub-task
>          Components: replication
>            Reporter: Jean-Daniel Cryans
>            Assignee: Lars Hofhansl
>         Attachments: 2195-v5.txt, 2195-v6.txt, 2195.txt
>
>
> We need to support cyclic replication by using the cluster id of each HlogKey and stop
replicating when it goes back to the original cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message