hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongjun Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6133) Make Balancer support exclude specified path
Date Fri, 14 Nov 2014 05:14:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211841#comment-14211841

Yongjun Zhang commented on HDFS-6133:

HI [~zhaoyunjiong],

Thanks for the good work here, and sorry for getting back late.

I reviewed the latest patch, and I think it indeed looks very close. I also think your answers
to Stack's question make sense to me. Couple of questions:

# The concept of favoredNodes pre-existed before your patch, now your patch defines that as
long as favoredNodes is passed, then pinning will be done. So we are changing the prior definition
of how favoredNodes are used. Why not add some additional interface to tell that pinning will
happen  so we have the option not to pin even if favoredNodes is passed? Not necessarily you
need to do what I suggested here, but I'd like to understand your thoughts here.
# Do we ever need interface to do unpinning?

And I have some comments, mostly nits.

* DFSOutputStream.java
Line 1413, change {{favoredNodes != null ? true: false}} to {{favoredNodes != null}}
* DataTransferProtocol.java, 
Line 95: "whether pinning the block" to "whether to pin the block"
* BlockReceiver.java
Line 178. Move {{this.pinning = pinning;" to before the debug msg printing block, and include
the pinning in th edebug message.
* FSDatasetSpi.java
Line 479: javadoc the setPinning /getPinning api
* FsDatasetImpl.java, Line 2618, javadoc that sticky bit is used for pinning purpose
* DataXceiver.java
Line 562, remove the newly added extra empty line
* datatransfer.proto
Line 125, need javadoc
* TestBalancer.java
Can you add a description what testBalancerWithPinnedBlocks does and what the expected result


> Make Balancer support exclude specified path
> --------------------------------------------
>                 Key: HDFS-6133
>                 URL: https://issues.apache.org/jira/browse/HDFS-6133
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: balancer & mover, namenode
>            Reporter: zhaoyunjiong
>            Assignee: zhaoyunjiong
>         Attachments: HDFS-6133-1.patch, HDFS-6133-2.patch, HDFS-6133-3.patch, HDFS-6133.patch
> Currently, run Balancer will destroying Regionserver's data locality.
> If getBlocks could exclude blocks belongs to files which have specific path prefix, like
"/hbase", then we can run Balancer without destroying Regionserver's data locality.

This message was sent by Atlassian JIRA

View raw message