hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6010) Make balancer able to balance data among specified servers
Date Thu, 06 Mar 2014 09:06:44 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13922183#comment-13922183

Yu Li commented on HDFS-6010:

Thanks [~devaraj] for the reply and CC Nicholas! 

Hi [~szetszwo], Thanks for taking a look here. I found your question similar with Das', so
I'd like to answer in one go. The background are described in HDFS-6009, allow me to quote

There're scenarios like mentioned in HBASE-6721 and HBASE-4210 that in multi-tenant deployments
of HBase we prefer to specify several groups of regionservers to serve different applications,
to achieve some kind of isolation or resource allocation. However, although the regionservers
are grouped, the datanodes which store the data are not, which leads to the case that one
datanode failure affects multiple applications, as we already observed in our product environment.

To relieve the above issue, we could take usage of the favored node feature (HDFS-2576) to
make regionserver able to locate data within its group, or say make datanodes also grouped
(passively), to form some level of isolation.

In this case, or any other case that needs datanodes to group, we would need a bunch of tools
to maintain the "group", including:
1. Making balancer able to balance data among specified servers, rather than the whole set
2. Set balance bandwidth for specified servers, rather than the whole set
3. Some tool to check whether the block is "cross-group" placed, and move it back if so

People might ask in this case why don't use phasically separated clusters, the answer would
be it's more convenient and saves people resource to manage one big cluster than several small

I also know there's other solution like HDFS-5776 to reduce the impact of bad datanode, but
I believe there're still scenarios which need more strict io isolation, so I think it's still
valuable to contribute our tools.

In case of undesirable moves caused by HBase compaction-like operation or replication caused
by disk-damage, we could supply tool like described in HDFS-6012 to check and move the "cross-group"
blocks back.

Let me know if any comments. :-)

> Make balancer able to balance data among specified servers
> ----------------------------------------------------------
>                 Key: HDFS-6010
>                 URL: https://issues.apache.org/jira/browse/HDFS-6010
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: balancer
>    Affects Versions: 2.3.0
>            Reporter: Yu Li
>            Assignee: Yu Li
>            Priority: Minor
>         Attachments: HDFS-6010-trunk.patch
> Currently, the balancer tool balances data among all datanodes. However, in some particular
case, we would need to balance data only among specified nodes instead of the whole set.
> In this JIRA, a new "-servers" option would be introduced to implement this.

This message was sent by Atlassian JIRA

View raw message