hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng Shao (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-611) Heartbeats times from Datanodes increase when there are plenty of blocks to delete
Date Tue, 03 Nov 2009 00:30:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772783#action_12772783
] 

Zheng Shao commented on HDFS-611:
---------------------------------

In light of some other discussions related to async deletion in JobTracker when it restarts,
I will rename BlockFileDeleter to AsyncDiskService, and move BlockFileDeleteTask inside the
FSDataset class. In this way, it will be easier to move AsyncDiskService to common and reuse
it in the JobTracker in the future.

> Traditionally hdfs did not distinguish between blocks and their replicas, which we found
very confusing while implementing append and tried to call new classes Replica*. So yes you
see a lot of Block* classes, but it would be really good to turn this in the right direction.
Wouldn't you agree that "replica" is a more precise term for a copy of a block on a specific
data-node.

I agree. My initial thought was that we should do that change in a single transaction for
everything, otherwise having both old and new conventions will make the things even more confusing.


> I think the abstraction should provide an api to delete replica files independently on
whether it is multi-threaded or single-threaded, so it makes sense to me to keep the implementation
details concealed in the deleter.

Users might need to know if an operation is sync or async. I will add a deleteAsync method.

> allowCoreThreadTimeOut 

Agree. I will add that.

> The key for the HashMap of threads is the reference to the volume

That makes sense. I will change the key of the HashMap to File, which represents the currentDir
of the FSVolume.


> Heartbeats times from Datanodes increase when there are plenty of blocks to delete
> ----------------------------------------------------------------------------------
>
>                 Key: HDFS-611
>                 URL: https://issues.apache.org/jira/browse/HDFS-611
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: dhruba borthakur
>            Assignee: Zheng Shao
>             Fix For: 0.20.2, 0.21.0, 0.22.0
>
>         Attachments: HDFS-611.branch-19.patch, HDFS-611.branch-19.v2.patch, HDFS-611.branch-20.patch,
HDFS-611.branch-20.v2.patch, HDFS-611.trunk.patch, HDFS-611.trunk.v2.patch, HDFS-611.trunk.v3.patch
>
>
> I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat
times from datanodes increase significantly from the normal value of 3 seconds to as large
as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially,
this causes the heartbeat times to increase.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message