Mailing-List: contact hbase-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Message-ID: <14549651.26461273214869777.JavaMail.jira@thor>
Date: Fri, 7 May 2010 02:47:49 -0400 (EDT)
From: "stack (JIRA)" <jira@apache.org>
To: hbase-issues@hadoop.apache.org
Subject: [jira] Updated: (HBASE-2413) Master does not respect generation
 stamps, may result in meta getting permanently offlined
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HBASE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-2413:
-------------------------

    Attachment: 2413-v7.patch

Attached is my v7 patch.  I've also posted it here, http://review.hbase.org/r/36/, into Todds fancy-pants new code review tool (which is excellent).  Please review.

> Master does not respect generation stamps, may result in meta getting permanently offlined
> ------------------------------------------------------------------------------------------
>
>                 Key: HBASE-2413
>                 URL: https://issues.apache.org/jira/browse/HBASE-2413
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.20.3
>            Reporter: Karthik Ranganathan
>            Assignee: stack
>             Fix For: 0.20.5
>
>         Attachments: 2413-v4.txt, 2413-v6.patch, 2413-v7.patch, newserver-v3.txt, newserver.txt
>
>
> This happens if the RS is restarted before the zk node expires. The sequence is as follows:
> 1. RS1 dies - lets say its server string was HOST1:PORT1:TS1
> 2. In a few seconds RS1 is restarted, it comes up as HOST1:PORT1:TS2 (TS2 is more recent than TS1)
> 3. Master gets a start up message from RS1 with the server name as HOST1:PORT1:TS2
> 4. Master adds this as a new RS, tries to red
> ---- The master does not use the generation stamps to detect that RS1 has already restarted.
> ---- Also, if RS1 contained meta, master would try to go to HOST1:PORT1:TS1. It would end up talking to HOST1:PORT1:TS2, which spews a bunch of not serving region exceptions.
> 5. zk node expires for HOST1:PORT1:TS1
> 6. Master tries to process shutdown for HOST1:PORT1:TS1 - this probably interferes with HOST1:PORT1:TS2  and ends up somehow removing the reassign meta in the master's queue.
> ---- Meta never comes online and master continues logging the following exception indefinitely:
> 2010-04-06 11:02:23,988 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose of test1,7094000000,1270220428234, false, reassign: true
> 2010-04-06 11:02:23,988 DEBUG org.apache.hadoop.hbase.master.ProcessRegionClose$1: Exception in RetryableMetaOperation:
> java.lang.NullPointerException
>         at org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:64)
>         at org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63)
>         at org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:494)
>         at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:429)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.