hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hsieh <...@cloudera.com>
Subject Re: HBase region server failure issues
Date Tue, 15 Apr 2014 15:26:30 GMT
Thanks for catching that -- it was a typo -- one wal per region.


On Tue, Apr 15, 2014 at 8:21 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> bq. In the case of an SSD world, it makes more sense to have one wal per
> node
>
> Was there a typo in the sentence above (one wal per node) ?
>
> Cheers
>
>
> On Tue, Apr 15, 2014 at 7:11 AM, Jonathan Hsieh <jon@cloudera.com> wrote:
>
> > It makes sense to have as many wals as # of spindles / replication factor
> > per machine.  This should be decoupled from the number of regions on a
> > region server.  So for a cluster with 12 spindles we should likely have
> at
> > least 4 wals (12 spindles / 3 replication factor), and need to do
> > experiments to see if going to 8 or some higher number makes sense (new
> wal
> > uses a disruptor pattern which avoids much contention on individual
> > writes).   So with your example, your 1000 regions would get sharded into
> > the 4 wals which would maximize io throughput, disk utilization, and
> reduce
> > time for recovery in the face of failure.
> >
> > In the case of an SSD world, it makes more sense to have one wal per node
> > once we have decent HSM support in HDFS.  The key win here will be in
> > recovery time -- if any RS goes down we only have to replay a regions
> edits
> > and not have to split or demux different region's edits.
> >
> > Jon.
> >
> >
> > On Mon, Apr 14, 2014 at 10:37 PM, Vladimir Rodionov
> > <vladrodionov@gmail.com>wrote:
> >
> > > Todd, how about 300 regions with 3x replication?  Or 1000 regions? This
> > is
> > > going to be 3000 files. on HDFS. per one RS. When I said that it does
> not
> > > scale, I meant that exactly that.
> > >
> >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // jon@cloudera.com // @jmhsieh
> >
>



-- 
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message