hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12282) Ozone: DeleteKey-4: Block delete via HB between SCM and DN
Date Tue, 22 Aug 2017 06:54:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136406#comment-16136406

Anu Engineer commented on HDFS-12282:

[~cheersyang] My concern comes from the issues I have seen in the HDFS world. The heartbeat
became a scalability bottleneck when it started doing too many things. Then HDFS eventually
had to invent things like "life-line protocol", which is really Heartbeat which does nothing
else. Disk I/O also has a very weird problem when the disk is slow or busy it will slow down
everything in its path. That in turn, makes the SCM think that data nodes are not HB-ing and
marks them as dead. 

This is a first-hand issue from very busy HDFS clusters, so life line protocol was developed
to purposefully make HB handling lightweight. I am just suggesting that we follow the model,
HB processing should be quick -- if you need to do any work based on HB, I think we should
post that to a queue and let an async system pick up the work.

> Ozone: DeleteKey-4: Block delete via HB between SCM and DN
> ----------------------------------------------------------
>                 Key: HDFS-12282
>                 URL: https://issues.apache.org/jira/browse/HDFS-12282
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, ozone, scm
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>         Attachments: Block delete via HB between SCM and DN.png, HDFS-12282-HDFS-7240.001.patch,
HDFS-12282-HDFS-7240.002.patch, HDFS-12282.WIP.patch
> This is the task 3 in the design doc, implements the SCM to datanode interactions. Including
> # SCM sends block deletion message via HB to datanode
> # datanode changes block state to deleting when processes the HB response
> # datanode sends deletion ACKs back to SCM
> # SCM handles ACKs and removes blocks in DB

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message