hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6133) Make Balancer support exclude specified path
Date Tue, 18 Nov 2014 18:45:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14216577#comment-14216577

Kihwal Lee commented on HDFS-6133:

We are in the process of optimizing {{BlockManager}} and block report processing. It will
be better to add the rest of the pieces involving reporting to NN and management on NN after
this change.  One thing that we need to make sure is that this jira is compatible with the
changes to come. Since DN has to include the pinning info in full and incremental block reports,
it needs to be recorded in memory, not queries against file system on demand.  The in-memory
state needs to be reconstructed on startup and things like {{DirectoryScanner}} might need
to be made aware of block pinning too. These don't need to be done in this jira, but the design
should not preclude such future changes.

For storing and reporting pinning information, we are planning to borrow some bits from the
block size variable. On the namenode side, the optimization will give us a few bits for storing
this information. I will file a separate jira when we have more details. This jira will be
linked as a dependency.

> Make Balancer support exclude specified path
> --------------------------------------------
>                 Key: HDFS-6133
>                 URL: https://issues.apache.org/jira/browse/HDFS-6133
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: balancer & mover, namenode
>            Reporter: zhaoyunjiong
>            Assignee: zhaoyunjiong
>         Attachments: HDFS-6133-1.patch, HDFS-6133-2.patch, HDFS-6133-3.patch, HDFS-6133.patch
> Currently, run Balancer will destroying Regionserver's data locality.
> If getBlocks could exclude blocks belongs to files which have specific path prefix, like
"/hbase", then we can run Balancer without destroying Regionserver's data locality.

This message was sent by Atlassian JIRA

View raw message