hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12648) DN should provide feedback to NN for throttling commands
Date Tue, 05 Dec 2017 22:57:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279323#comment-16279323

Daryn Sharp commented on HDFS-12648:

I think we need this support before HDFS-12647.  It doesn't make much sense for that jira
to not send heartbeats when the queue is full.

I think the simplest answer is the DN heartbeats the max number of commands it will accept.
 The NN simply hands out no more than that.  Replication commands take priority over invalidation
work.  The former implies data is at risk, the former is just trying to clean up space.  Work
is already computed in that order anyway.

The max commands can default to Integer.MAX_VALUE for backwards compatibility with old nodes
since that's effectively what it is today.  Then when HDFS-12647 makes the DN smarter, we'll
have the NN support already in place.

> DN should provide feedback to NN for throttling commands
> --------------------------------------------------------
>                 Key: HDFS-12648
>                 URL: https://issues.apache.org/jira/browse/HDFS-12648
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: 2.8.0
>            Reporter: Daryn Sharp
>            Assignee: Hanisha Koneru
> The NN should avoid sending commands to a DN with a high number of outstanding commands.
 The heartbeat could provide this feedback via perhaps a simple count of the commands or rate
of processing.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message