hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5209) HConnection/HMasterInterface should allow for way to get hostname of currently active master in multi-master HBase setup
Date Wed, 07 Mar 2012 05:48:01 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223994#comment-13223994
] 

jiraposter@reviews.apache.org commented on HBASE-5209:
------------------------------------------------------



bq.  On 2012-03-07 00:53:15, jmhsieh wrote:
bq.  > Should probably file follow-on issues to update AvroUtil.csToACS (convert to ClusterStatus
to Avro for avro server) and likely to thrift/rest clients as well.

OK.


bq.  On 2012-03-07 00:53:15, jmhsieh wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/ClusterStatus.java, line 234
bq.  > <https://reviews.apache.org/r/3892/diff/3/?file=75401#file75401line234>
bq.  >
bq.  >     Would a copy be safer?

backupMasters has no way of being modified after the instance is created, so I don't think
a copy is any safer, if you are thinking about "safer" == "immutable".  If someone added a
mutator method for backupMasters, then yes we would have a problem, but I find that unlikely
in this case given the mission of ClusterStatus being a snapshot of the cluster state.


bq.  On 2012-03-07 00:53:15, jmhsieh wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/ClusterStatus.java, line 180
bq.  > <https://reviews.apache.org/r/3892/diff/3/?file=75401#file75401line180>
bq.  >
bq.  >     Since the type is Collection<xxx>, I have a feeling that there may be
a equality and hash issues here. This should probably be:
bq.  >     
bq.  >     this.backupMasters.containsAll(((ClusterStatus)o).backupMasters)
bq.  >     
bq.  >     alternately, (if not present), add unit tests to convince that equality and
hash operations work?

Easier to just change this to containsAll() rather than add unit tests, since I think another
patch is needed here.


bq.  On 2012-03-07 00:53:15, jmhsieh wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/ClusterStatus.java, line 70
bq.  > <https://reviews.apache.org/r/3892/diff/3/?file=75401#file75401line70>
bq.  >
bq.  >     Unlikely, but someone could possibly name their host "unknown".  Maybe change
this to be something that is normally illegal as a host name (start with a symbol, maybe "#unknown#".)
 
bq.  >

OK.


bq.  On 2012-03-07 00:53:15, jmhsieh wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 1387
bq.  > <https://reviews.apache.org/r/3892/diff/3/?file=75403#file75403line1387>
bq.  >
bq.  >     Is this guaranteed to read/return values in the same order all the time?  This
may be is important for the equality comparison and hashcode comparisons. (Doesn't seemed
to be used in hbase core code, but exposed to clients).

I'll add a sort with appropriate Comparator, as the backing function from ZooKeeper does not
guarantee ordering.


bq.  On 2012-03-07 00:53:15, jmhsieh wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java, line 143
bq.  > <https://reviews.apache.org/r/3892/diff/3/?file=75402#file75402line143>
bq.  >
bq.  >     Would this create a small race -- why not have all masters write into a "potential-masters"
zdir instead of "backup-masters" zdir and then juat have a node for the active master?
bq.  >     
bq.  >     Then we don't have to worry about someone accidentally reading an in between
state where the a master is active and still in backup-masters or not master and not in backup-masters.

I think you are suggesting having a list of potential masters, and deriving the list of backup
masters from that list by just subtracting the active master.

The problem then is that there is then a race whenever you derive the backup masters, between
when you read the potential masters and when you read the active master, as those are separate
operations and therefore something can change in between them.  You're just moving where the
race happens.

What is really needed here is a read-write lock that protects this area.  There are no read-write
(or any) ZK-based locks currently in HBase, so I'm not sure if we want to add an implementation
of that just for this case.


- David


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3892/#review5673
-----------------------------------------------------------


On 2012-02-16 06:30:31, David Wang wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3892/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2012-02-16 06:30:31)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Problem:
bq.  There is no method in the HBase client-facing APIs to determine which of the masters
is currently active.  This can be especially useful in setups with multiple backup masters.
bq.  
bq.  Solution:
bq.  Augment ClusterStatus to return the currently active master and the list of backup masters.
bq.  
bq.  Notes:
bq.  * I uncovered a race condition in ActiveMasterManager, between when it determines that
it did not win the original race to be the active master, and when it reads the ServerName
of the active master.  If the active master goes down in that time, the read to determine
the active master's ServerName will fail ungracefully and the candidate master will abort.
 The solution incorporated in this patch is to check to see if the read of the ServerName
succeeded before trying to use it.
bq.  * I fixed some minor formatting issues while going through the code.  I can take these
changes out if it is considered improper to commit such non-related changes with the main
changes.
bq.  
bq.  
bq.  This addresses bug HBASE-5209.
bq.      https://issues.apache.org/jira/browse/HBASE-5209
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    src/main/java/org/apache/hadoop/hbase/ClusterStatus.java b849429 
bq.    src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java 2f60b23 
bq.    src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9d21903 
bq.    src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java f6f3f71 
bq.    src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 111f76e 
bq.    src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 3e3d131 
bq.    src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java 16e4744 
bq.    src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java bc98fb0 
bq.  
bq.  Diff: https://reviews.apache.org/r/3892/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  * Ran mvn -P localTests test multiple times - no new tests fail
bq.  * Ran mvn -P localTests -Dtest=TestActiveMasterManager test multiple runs - no failures
bq.  * Ran mvn -P localTests -Dtest=TestMasterFailover test multiple runs - no failures
bq.  * Started active and multiple backup masters, then killed active master, then brought
it back up (will now be a backup master)
bq.    * Did the following before and after killing
bq.      * hbase hbck -details - checked output to see that active and backup masters are
reported properly
bq.      * zk_dump - checked that active and backup masters are reported properly
bq.  * Started cluster with no backup masters to make sure change operates correctly that
way
bq.  * Tested build with this diff vs. build without this diff, in all combinations of client
and server
bq.    * Verified that new client can run against old servers without incident and with the
defaults applied.
bq.    * Note that old clients get an error when running against new servers, because the
old readFields() code in ClusterStatus does not handle exceptions of any kind.  This is not
solvable, at least in the scope of this change.
bq.  
bq.  12/02/15 15:15:38 INFO zookeeper.ClientCnxn: Session establishment complete on server
haus02.sf.cloudera.com/172.29.5.33:30181, sessionid = 0x135834c75e20008, negotiated timeout
= 5000
bq.  12/02/15 15:15:39 ERROR io.HbaseObjectWritable: Error in readFields
bq.  A record version mismatch occured. Expecting v2, found v3
bq.          at org.apache.hadoop.io.VersionedWritable.readFields(VersionedWritable.java:46)
bq.          at org.apache.hadoop.hbase.ClusterStatus.readFields(ClusterStatus.java:247)
bq.          at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:583)
bq.          at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
bq.  
bq.  * Ran dev-support/test-patch.sh - no new issues fail:
bq.  
bq.  -1 overall.
bq.  
bq.      +1 @author.  The patch does not contain any @author tags.
bq.  
bq.      +1 tests included.  The patch appears to include 7 new or modified tests.
bq.  
bq.      -1 javadoc.  The javadoc tool appears to have generated -136 warning messages.  
bq.  
bq.      +1 javac.  The applied patch does not increase the total number of javac compiler
warnings.
bq.  
bq.      +1 findbugs.  The patch does not introduce any new Findbugs (version ) warnings.
bq.  
bq.      +1 release audit.  The applied patch does not increase the total number of release
audit warnings.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  David
bq.  
bq.


                
> HConnection/HMasterInterface should allow for way to get hostname of currently active
master in multi-master HBase setup
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-5209
>                 URL: https://issues.apache.org/jira/browse/HBASE-5209
>             Project: HBase
>          Issue Type: Improvement
>          Components: master
>    Affects Versions: 0.90.5, 0.92.0, 0.94.0
>            Reporter: Aditya Acharya
>            Assignee: David S. Wang
>             Fix For: 0.92.1, 0.94.0
>
>         Attachments: 5209.addendum, HBASE_5209_v5.diff
>
>
> I have a multi-master HBase set up, and I'm trying to programmatically determine which
of the masters is currently active. But the API does not allow me to do this. There is a getMaster()
method in the HConnection class, but it returns an HMasterInterface, whose methods do not
allow me to find out which master won the last race. The API should have a getActiveMasterHostname()
or something to that effect.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message