hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Barr <dave.b...@gmail.com>
Subject Re: hbase and dfs balancer
Date Tue, 05 Jun 2012 19:03:46 GMT
You kind of don't have a choice.  Running out of space in a datanode
is Very Bad.

We run an HDFS balance on a semi regular basis, whenever I do a bunch
of node replacements/rebuilds or additions.  Also over time as entropy
builds and we do data cleanups.

You do lose locality, but there's an easy to fix, namely to just do a
major compaction.  When the RS rewrites the datafile it will be local
again. Also, as you do flushes and such, data will slowly become local

On Tue, Jun 5, 2012 at 11:29 AM, Eric Raymond <eraymond@mochimedia.com> wrote:
> Hello all,
> I am not sure this goes here or not, but I wanted to get a formal answer on
> using the hdfs balancer with hbase.  From what I heard a while back, its
> not recommended to run alongside hbase, as it can destroy the regions.  How
> can I rebalance my datanode/dfs cluster as, it doesnt seem to do this
> automatically?  I have dfs.balance.bandwidthPerSec set to 100Mbit, but I am
> seeing that a majority of the nodes are reaching 99% full, while 3-4 others
> are only at 30%.
> Can anyone advise on this?
> Thanks,
> Eric

View raw message