Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 31 Mar 2011 05:12:05 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: 
 <1691142426.23426.1301548325686.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HBASE-3071) Graceful decommissioning of a
 regionserver
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HBASE-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-3071:
-------------------------

    Attachment: 3701-v3.txt

Added more retries of move (saw case where server wasn't fully up yet so move to that server failed and on retry, we put it back in old location.. which looked like it'd never moved).

Tested under load.  All just runs slower.  But loading keeps going.

More notes.

So, we unload a region at a time till zero.  Then we do clean shutdown.  This triggers over in master the recover of an expired server only there are no logs to process because it had a clean shutdown, so the server comes up pretty fast and is then ready to take on regions again.

If balancer runs, it doesn't cause 'failure' but what it does is that while unloading, it can add some new regions to a server.  We'll then move those off.  Now, this RS will have the old burden plus the new associated with it.  On replay the cluster will be out of kilter, off-balance... so its kinda important having the cluster basically balanced before running this script.

If exception processing a server, we'll skip it.  It can be hard to figure a failure if you don't keep logs and look at them after.  The region mover logs each region move which can be a bunch and if server 50 of 100 failed, you could have a cluster that was 99% upgraded, only.

A wrapper script that runs the rolling restart -- enable/disabling balancer, upping and downing masters (see current bin/rolling_restart.sh) -- we could check zk to see if a servers' servername changed (servername is hostname + port + startcode).  If it hadn't, we could on the end flag it as failed rolling restart.

I'm figuring wrapper script outside the scope of this issue.  For now I do a manual enable/disable of balancer and then do a for loop calling bin/graceful_stop.sh...

> Graceful decommissioning of a regionserver
> ------------------------------------------
>
>                 Key: HBASE-3071
>                 URL: https://issues.apache.org/jira/browse/HBASE-3071
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: stack
>         Attachments: 3071.txt, 3701-v2.txt, 3701-v3.txt
>
>
> Currently if you stop a regionserver nicely, it'll put up its stopping flag and then close all hosted regions.  While the stopping flag is in place all region requests are rejected.  If this server was under load, closing could take a while.  Only after all is closed is the master informed and it'll restart assigning (in old master, master woud get a report with list of all regions closed, in new master the zk expired is triggered and we'll run shutdown handler).
> At least in new master, we have means of disabling balancer, and then moving the regions off the server one by one via HBaseAdmin methods -- we shoud write a script to do this at least for rolling restarts -- but we need something better.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira