[ https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310564#comment-14310564
]
Benoy Antony commented on HDFS-6133:
------------------------------------
I know that there are different ways to accomplish the desired functionality of pinning the
blocks under a directory to a set of nodes.
I want to suggest yet another approach using storage policies.
This is based on the fact that the set of favored nodes can be considered as a tier.
Let’s say blocks under /hbase needs to be pinned to a set of nodes.
1. Mark the data.dirs on those set of datanodes as a special storage type , say HBASE_TIER.
[HBASE_TIER]/hadoop01/data
2. Define a storage policy - HBASE_POLICY which insists that all blocks following the policy,
reside in HBASE_TIER
3. Set storage policy of /hbase to HBASE_POLICY
Since the balancer(and mover) considers the storage policy while moving/balancing, balancer
will not move these blocks out of the favored nodes(HBASE_TIER)
Now if we want to keep other blocks ( non hbase blocks) on these datanodes as well(may not
be a great idea), then we can model it by having multiple storage types on a single directory
like [HBASE_TIER,DISK]/hadoop01/data. This make sense since one want to have blocks having
multiple storage characteristics on the same node.
This will need a (minor) code modification.
> Make Balancer support exclude specified path
> --------------------------------------------
>
> Key: HDFS-6133
> URL: https://issues.apache.org/jira/browse/HDFS-6133
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, namenode
> Reporter: zhaoyunjiong
> Assignee: zhaoyunjiong
> Attachments: HDFS-6133-1.patch, HDFS-6133-2.patch, HDFS-6133-3.patch, HDFS-6133-4.patch,
HDFS-6133-5.patch, HDFS-6133-6.patch, HDFS-6133-7.patch, HDFS-6133-8.patch, HDFS-6133.patch
>
>
> Currently, run Balancer will destroying Regionserver's data locality.
> If getBlocks could exclude blocks belongs to files which have specific path prefix, like
"/hbase", then we can run Balancer without destroying Regionserver's data locality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
|