hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-339) Periodically move blocks from full nodes to those with space
Date Sun, 17 Jul 2011 18:41:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Harsh J resolved HDFS-339.

    Resolution: Not A Problem

The introduction of the DFS Balancer (post '06, apparently) provides the feature of moving
blocks around and balancing the DFS DNs.

I personally do not think its a good idea to add a monitor to the NN for auto-triggering balancers
since it would use up bandwidth without the user/admin ever knowing about it. One could surely
write external tools to achieve this monitoring and running separately though.

Resolving a Not-A-Problem, but do reopen if you feel strongly that the NN would really benefit
from an additional service as this.

>  Periodically move blocks from full nodes to those with space
> -------------------------------------------------------------
>                 Key: HDFS-339
>                 URL: https://issues.apache.org/jira/browse/HDFS-339
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Bryan Pendleton
>            Assignee: Sameer Paranjpye
> Continuance of Hadoop-386. The patch to that issue makes it possible to redistribute
blocks (change replication up, wait for replication to succeed, then lower replication again).
However, this requires a lot more space, is not automatic, and doesn't respect a reasonable
I/O limit. I have actually had MapReduce jobs fail from block missing execptions after having
recently changed the replication level (from 3 to 4, with no underreplications to start with)
because the datanodes were too slow responding to requests while performing the necessary
> A good fix to this problem would be a low-priority thread on the NameNode that schedules
low-priority replications of blocks on over-full machines, followed by the removal of the
extra replications. It might be worth having a specific prototocol for asking for these low-priority
copies to happen in the datanodes, so that they continue to service (and be available to service)
normal block requests.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message