hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
Date Thu, 10 Sep 2009 23:07:57 GMT

    [ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753883#action_12753883
] 

Doug Cutting commented on HDFS-611:
-----------------------------------

> Why is the heartbeat thread involved with block deletion?

The namenode returns lists of blocks to delete in heartbeat responses.

>  The Nameode will think that the datanode has this block whereas actually it is now gone
from the datanode.

The datanode might remove blocks immediately from its in-memory lists that are used to generate
block reports but the background thread could remove the actual block files.  If the datanode
crashes then the discrepancy between what's on disk and what the namenode thinks should be
on disk will be resolved at restart when it sends its initial block report.  Could that work?

> Heartbeats times from Datanodes increase when there are plenty of blocks to delete
> ----------------------------------------------------------------------------------
>
>                 Key: HDFS-611
>                 URL: https://issues.apache.org/jira/browse/HDFS-611
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>
> I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat
times from datanodes increase significantly from the normal value of 3 seconds to as large
as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially,
this causes the heartbeat times to increase.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message