hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikael Sitruk <mikael.sit...@gmail.com>
Subject Re: Server Crash Recovery (WAS --> Re: ANN: The third hbase 0.94.0 release candidate is available for download)
Date Wed, 16 May 2012 07:25:10 GMT
Hey St.Ack

Thanks for clarifications,

For 4. replay of log: (Please correct if i'm wrong)
So the RS will:
a. split the log via HLogSplitter, write concurrently log content to other
log files under each region,
b. replay those smaller logs into its own memstore and own logs (is it done
when the region becomes on-line?)
During this replay the RS may be subject to log flushing, and to compaction
(flush will create more store file that will reach the min/max compaction),
and other regular background task.

During this time (log replay) the RS also accepts other client requests (on
its own regions, not the ones it got assigned) or they are blocked?
In case the RS is handling other requests too, is there any priority for
WAL replay?


On Tue, May 15, 2012 at 4:11 PM, Stack <stack@duboce.net> wrote:

> (Moved this conversation off the vote thread)
> On Sat, May 12, 2012 at 3:14 PM, Mikael Sitruk <mikael.sitruk@gmail.com>
> wrote:
> > So in case a RS goes down, the master will split the log and reassign the
> > regions to other RS, then each RS will replay the log, during this step
> the
> > regions are unavailable, and clients will got exceptions.
> To be clear, the log splitting will result in each region having under
> it, its own edits only for replay.
> > 1. how the master will choose a RS to assign a region?
> Random currently.  It picks from the list of currently live RSs.
> > 2. how many RS will be involved in this reassignment
> All remaining live regionservers.
> > 3. client that got exception should renew their connections or they can
> > reuse the same one?
> When client gets the NotServingRegionException, it goes back to the
> .META. to find location of the region.  It'll then retry this
> location.   The location maybe the still-down server.  Client will
> keep at this until it either timesout or it the address is updated in
> .META. with the new location.
> > 4. is there a way to figure out how long this split+replay will take
> > (either by formula at the design time of a deployment, or at runtime via
> > API asking the master for example)???
> >
> Usually its a factor of how many WAL files the regionserver was
> carrying when it went down (You'll see in the logs where we sometimes
> force flushes to clear the memstores carrying oldest edits just so we
> can clear out old WAL files.  The log roller figures out what needs
> flushing.  See
> http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/LogRoller.html#95
> ).
>  You can set the max number of WALs a regionserver carries; grep
> 'hbase.regionserver.maxlogs' (We don't seem to doc this one -- we
> should fix that).
> St.Ack

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message