hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stu24m...@yahoo.com
Subject Re: keeping an active hdfs cluster balanced
Date Thu, 17 Mar 2011 22:15:26 GMT
Hello Ted,

I have a small, 8 DN cluster, 6 of which are regionservers. Some have 3TB, others have 2TB.
All have all disks available to hdfs - including the OS/system disk :|

The majority of the data goes to HBase, which then writes to hdfs. Some data is written to
hdfs via thrift.

Take care,
-----Original Message-----
From: Ted Dunning <tdunning@maprtech.com>
Date: Thu, 17 Mar 2011 14:23:10 
To: <hdfs-user@hadoop.apache.org>
Reply-To: hdfs-user@hadoop.apache.org
Cc: Stuart Smith<stu24mail@yahoo.com>
Subject: Re: keeping an active hdfs cluster balanced

How large a cluster?

How large is each data-node?  How much disk is devoted to hbase?

How does your HDFS data arrive?  From one or a few machines in the cluster?
 From outside the cluster?

On Thu, Mar 17, 2011 at 12:13 PM, Stuart Smith <stu24mail@yahoo.com> wrote:

> Parts of this may end up on the hbase list, but I thought I'd start here.
> My basic problem is:
> My cluster is getting full enough that having one data node go down does
> put a bit of pressure on the system (when balanced, every DN is more than
> half full).
> I write (and delete) pretty actively to Hbase & some hdfs direct.
> The cluster keeps drifting dangerously out of balance.
> I run the balancer daily, but:
>   - I've seen reports that you shouldn't rebalance with regionservers
> running, yet, I don't really have a choice. Without HBase, my system is
> pretty much down. If it gets out of balance, it will also come down.
>  Anybody here have any idea how badly running the balancer on a heavily
> active system messes things up? (for hdfs/hbase - if anyone knows).
>   - Possibly somewhat related: I'm seeing more "failed to move block"
> errors in my balancer logs. It got to the point were I wasn't seeing any
> effective rebalancing occur. I've turned off access to the cluster and
> rebalanced (one node was down to 10% free space, a couple others when up to
> 50 or more). I'm back down to around 20-40% free space on each node (as
> reported by the hdfs web interface).
>    How effective is the balancer on a active cluster? Is there any way to
> make it's life easier, so it can stay in balance with daily runs?
> I'm not sure why the one node ends up being so heavily favored, either. The
> favoritism even seems to survive taking the node down, and bringing it back
> up. If I can't find the resources to upgrade, I might try that again, but
> I'm less than hopeful about it.
> Any ideas? Or do I just need better hardware? Not sure if that's an option,
> though..
> Take care,
>  -stu

View raw message