hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benoy Antony (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6133) Make Balancer support exclude specified path
Date Sat, 07 Feb 2015 05:23:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310564#comment-14310564
] 

Benoy Antony commented on HDFS-6133:
------------------------------------

I know that there are different ways to accomplish the desired functionality of pinning the
blocks under a directory to a set of nodes.
I want to suggest yet another approach using storage policies.

This is based on the fact that the set of favored nodes can be considered as a tier. 

Let’s say blocks under /hbase needs to be pinned to a set of nodes.

1. Mark the data.dirs on those set of datanodes as a special storage type , say HBASE_TIER.
[HBASE_TIER]/hadoop01/data
2. Define a storage policy - HBASE_POLICY which insists that all blocks following the policy,
reside in HBASE_TIER
3. Set storage policy of /hbase to HBASE_POLICY 

Since the balancer(and mover) considers the storage policy while moving/balancing, balancer
will not move these blocks out of the favored nodes(HBASE_TIER)

Now if we want to keep other blocks ( non hbase blocks) on these datanodes as well(may not
be a great idea), then we can model it by having multiple storage types on a single directory
like [HBASE_TIER,DISK]/hadoop01/data. This make sense since one want to have blocks having
multiple storage characteristics on the same node.
This will need a (minor) code modification.


> Make Balancer support exclude specified path
> --------------------------------------------
>
>                 Key: HDFS-6133
>                 URL: https://issues.apache.org/jira/browse/HDFS-6133
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: balancer & mover, namenode
>            Reporter: zhaoyunjiong
>            Assignee: zhaoyunjiong
>         Attachments: HDFS-6133-1.patch, HDFS-6133-2.patch, HDFS-6133-3.patch, HDFS-6133-4.patch,
HDFS-6133-5.patch, HDFS-6133-6.patch, HDFS-6133-7.patch, HDFS-6133-8.patch, HDFS-6133.patch
>
>
> Currently, run Balancer will destroying Regionserver's data locality.
> If getBlocks could exclude blocks belongs to files which have specific path prefix, like
"/hbase", then we can run Balancer without destroying Regionserver's data locality.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message