hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Ranganathan (JIRA)" <j...@apache.org>
Subject [jira] Created: (HBASE-2413) Master does not respect generation stamps, may result in meta getting permanently offlined
Date Tue, 06 Apr 2010 22:21:33 GMT
Master does not respect generation stamps, may result in meta getting permanently offlined

                 Key: HBASE-2413
                 URL: https://issues.apache.org/jira/browse/HBASE-2413
             Project: Hadoop HBase
          Issue Type: Bug
          Components: master
    Affects Versions: 0.20.3
            Reporter: Karthik Ranganathan

This happens if the RS is restarted before the zk node expires. The sequence is as follows:

1. RS1 dies - lets say its server string was HOST1:PORT1:TS1
2. In a few seconds RS1 is restarted, it comes up as HOST1:PORT1:TS2 (TS2 is more recent than
3. Master gets a start up message from RS1 with the server name as HOST1:PORT1:TS2
4. Master adds this as a new RS, tries to red
**** The master does not use the generation stamps to detect that RS1 has already restarted.
**** Also, if RS1 contained meta, master would try to go to HOST1:PORT1:TS1. It would end
up talking to HOST1:PORT1:TS2, which spews a bunch of not serving region exceptions.
5. zk node expires for HOST1:PORT1:TS1
6. Master tries to process shutdown for HOST1:PORT1:TS1 - this probably interferes with HOST1:PORT1:TS2
 and ends up somehow removing the reassign meta in the master's queue.
**** Meta never comes online and master continues logging the following exception indefinitely:

2010-04-06 11:02:23,988 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose
of test1,7094000000,1270220428234, false, reassign: true
2010-04-06 11:02:23,988 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception
in RetryableMetaOperation:
        at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:64)
        at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63)
        at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:494)
        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:429)

Fix is to have master recognize the RS1 restart before the zk node expires and handle it right
away. There should only be one entry for each HOST:PORT - the one with the latest TS.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message