hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-3071) Graceful decommissioning of a regionserver
Date Thu, 31 Mar 2011 05:12:05 GMT

     [ https://issues.apache.org/jira/browse/HBASE-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-3071:
-------------------------

    Attachment: 3701-v3.txt

Added more retries of move (saw case where server wasn't fully up yet so move to that server
failed and on retry, we put it back in old location.. which looked like it'd never moved).

Tested under load.  All just runs slower.  But loading keeps going.

More notes.

So, we unload a region at a time till zero.  Then we do clean shutdown.  This triggers over
in master the recover of an expired server only there are no logs to process because it had
a clean shutdown, so the server comes up pretty fast and is then ready to take on regions
again.

If balancer runs, it doesn't cause 'failure' but what it does is that while unloading, it
can add some new regions to a server.  We'll then move those off.  Now, this RS will have
the old burden plus the new associated with it.  On replay the cluster will be out of kilter,
off-balance... so its kinda important having the cluster basically balanced before running
this script.

If exception processing a server, we'll skip it.  It can be hard to figure a failure if you
don't keep logs and look at them after.  The region mover logs each region move which can
be a bunch and if server 50 of 100 failed, you could have a cluster that was 99% upgraded,
only.

A wrapper script that runs the rolling restart -- enable/disabling balancer, upping and downing
masters (see current bin/rolling_restart.sh) -- we could check zk to see if a servers' servername
changed (servername is hostname + port + startcode).  If it hadn't, we could on the end flag
it as failed rolling restart.

I'm figuring wrapper script outside the scope of this issue.  For now I do a manual enable/disable
of balancer and then do a for loop calling bin/graceful_stop.sh...

> Graceful decommissioning of a regionserver
> ------------------------------------------
>
>                 Key: HBASE-3071
>                 URL: https://issues.apache.org/jira/browse/HBASE-3071
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: stack
>         Attachments: 3071.txt, 3701-v2.txt, 3701-v3.txt
>
>
> Currently if you stop a regionserver nicely, it'll put up its stopping flag and then
close all hosted regions.  While the stopping flag is in place all region requests are rejected.
 If this server was under load, closing could take a while.  Only after all is closed is the
master informed and it'll restart assigning (in old master, master woud get a report with
list of all regions closed, in new master the zk expired is triggered and we'll run shutdown
handler).
> At least in new master, we have means of disabling balancer, and then moving the regions
off the server one by one via HBaseAdmin methods -- we shoud write a script to do this at
least for rolling restarts -- but we need something better.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message