hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space
Date Tue, 21 Jul 2015 20:47:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14635788#comment-14635788

Anu Engineer commented on HDFS-3570:

Hi [~ajisakaa] Thanks for the patch. I think it is a good change and defines the notion of
remaining space is a better way. However I had a minor question. In {{DatanodeInfo.getBlockPoolUsedPercent}}
and {{Datanodeinfo.getRemainingPercent}} I see we are still referring to {{capacity}} in the
old sense. Would that mean now we have two different views of what is the available capacity
of a DN; That is, one that is viewed from the balancer vs. something like {{DFSClient.getDatanodeReport}}.

> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space
> --------------------------------------------------------------------------------
>                 Key: HDFS-3570
>                 URL: https://issues.apache.org/jira/browse/HDFS-3570
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: balancer & mover
>    Affects Versions: 2.0.0-alpha
>            Reporter: Harsh J
>            Assignee: Akira AJISAKA
>            Priority: Minor
>         Attachments: HDFS-3570.2.patch, HDFS-3570.aash.1.patch
> Report from a user here: https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among dfs.data.dirs, and very
little DFS usage (which is computed against total possible capacity). 
> Balancer apparently only looks at the usage, and ignores to consider that non-DFS usage
may also be high on a DN/cluster. Hence, it thinks that if a DFS Usage report from DN is 8%
only, its got a lot of free space to write more blocks, when that isn't true as shown by the
case of this user. It went on scheduling writes to the DN to balance it out, but the DN simply
can't accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on {{(100-(actual
remaining space))/(capacity)}}, as opposed to the current {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space being used for
non DN data, but it does expose a valid bug.

This message was sent by Atlassian JIRA

View raw message