hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng Shao (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
Date Wed, 04 Nov 2009 04:20:32 GMT

     [ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zheng Shao updated HDFS-611:
----------------------------

    Attachment: HDFS-611.trunk.v5.patch


bq. It would look really good if you could move the deleteAsync() method together with the
static class ReplicaFileDeleteClass inside AsyncDiskService. I understand your motivation
that you want to keep AsyncDiskService independent of data-node connotations if the goal is
to reuse it in MR, but this could be done by simple factoring our the common base class if
such usage in MR will ever materialize.

For the MR usage, I am think about moving the AsyncDiskService class directly to common. Aggregation
seems better than inheritance here, just like ThreadPool (I guess we don't extend ThreadPoolExecutor
for different types of Tasks).

It seems to me that AsyncDiskService should not need to have knowledge of whatever Task is
requested by the caller. In particular, the decDfsUsage call in the ReplicaFileDeletionTask
is closely related to FSDataSet, and should be maintain inside FSDataSet. What do you think?

bq. A nit: could you convert the comment for AsyncDiskService to JavaDoc comment by just adding
one more star.

Added the missing "*" for javadoc.


> Heartbeats times from Datanodes increase when there are plenty of blocks to delete
> ----------------------------------------------------------------------------------
>
>                 Key: HDFS-611
>                 URL: https://issues.apache.org/jira/browse/HDFS-611
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: dhruba borthakur
>            Assignee: Zheng Shao
>             Fix For: 0.20.2, 0.21.0, 0.22.0
>
>         Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch,
HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch,
HDFS-611.trunk.v4.patch, HDFS-611.trunk.v5.patch
>
>
> I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat
times from datanodes increase significantly from the normal value of 3 seconds to as large
as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially,
this causes the heartbeat times to increase.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message