hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-339) Periodically move blocks from full nodes to those with space
Date Sun, 17 Jul 2011 18:41:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Harsh J resolved HDFS-339.
--------------------------

    Resolution: Not A Problem

The introduction of the DFS Balancer (post '06, apparently) provides the feature of moving
blocks around and balancing the DFS DNs.

I personally do not think its a good idea to add a monitor to the NN for auto-triggering balancers
since it would use up bandwidth without the user/admin ever knowing about it. One could surely
write external tools to achieve this monitoring and running separately though.

Resolving a Not-A-Problem, but do reopen if you feel strongly that the NN would really benefit
from an additional service as this.

>  Periodically move blocks from full nodes to those with space
> -------------------------------------------------------------
>
>                 Key: HDFS-339
>                 URL: https://issues.apache.org/jira/browse/HDFS-339
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Bryan Pendleton
>            Assignee: Sameer Paranjpye
>
> Continuance of Hadoop-386. The patch to that issue makes it possible to redistribute
blocks (change replication up, wait for replication to succeed, then lower replication again).
However, this requires a lot more space, is not automatic, and doesn't respect a reasonable
I/O limit. I have actually had MapReduce jobs fail from block missing execptions after having
recently changed the replication level (from 3 to 4, with no underreplications to start with)
because the datanodes were too slow responding to requests while performing the necessary
replications.
> A good fix to this problem would be a low-priority thread on the NameNode that schedules
low-priority replications of blocks on over-full machines, followed by the removal of the
extra replications. It might be worth having a specific prototocol for asking for these low-priority
copies to happen in the datanodes, so that they continue to service (and be available to service)
normal block requests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message