Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Message-ID: <1486377888.1257304952440.JavaMail.jira@brutus>
Date: Wed, 4 Nov 2009 03:22:32 +0000 (UTC)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Subject: [jira] Commented: (HDFS-611) Heartbeats times from Datanodes
 increase when there are plenty of blocks to delete
In-Reply-To: <270305594.1252570077770.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773368#action_12773368 ] 

Konstantin Shvachko commented on HDFS-611:
------------------------------------------

I like AsyncDiskService. Not sure about MR usage but we can definitely add more data-node operations under the class that require async operations.
Two things:
# It would look really good if you could move the deleteAsync() method together with the static class ReplicaFileDeleteClass inside AsyncDiskService. I understand your motivation that you want to keep AsyncDiskService independent of data-node connotations if the goal is to reuse it in MR, but this could be done by simple factoring our the common base class if such usage in MR will ever materialize.
# A nit: could you convert the comment for AsyncDiskService to JavaDoc comment by just adding one more star.

Other than that the patch looks good. Test failure is related to HDFS-733.

> Heartbeats times from Datanodes increase when there are plenty of blocks to delete
> ----------------------------------------------------------------------------------
>
>                 Key: HDFS-611
>                 URL: https://issues.apache.org/jira/browse/HDFS-611
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: dhruba borthakur
>            Assignee: Zheng Shao
>             Fix For: 0.20.2, 0.21.0, 0.22.0
>
>         Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch, HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch, HDFS-611.trunk.v4.patch
>
>
> I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.