hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-7270) Implementing congestion control in writing pipeline
Date Fri, 14 Nov 2014 01:35:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211668#comment-14211668
] 

Haohui Mai edited comment on HDFS-7270 at 11/14/14 1:34 AM:
------------------------------------------------------------

The v1 patch introduces a mechanism so that a DN can flag itself as congested. There is no
policy implemented in this patch.

The implementation is based on the idea proposed in RFC 3168. It takes the 6th and the 7th
bits on the pipeline ack to flag the state of the congestion information, where there is zero
overhead in the wire format. It is also wire-compatible with the old versions.


was (Author: wheat9):
The v1 patch introduces a mechanism so that a DN can flag itself as congested. There is no
policy implemented in this patch.

The implementation is based on the idea proposed in RFC 3168. It takes the 6th and the 7th
bits on the pipeline ack to flag the state of the congestion information. It is wire compatible
with the old versions.

> Implementing congestion control in writing pipeline
> ---------------------------------------------------
>
>                 Key: HDFS-7270
>                 URL: https://issues.apache.org/jira/browse/HDFS-7270
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>            Reporter: Haohui Mai
>            Assignee: Haohui Mai
>         Attachments: HDFS-7270.000.patch
>
>
> When a client writes to HDFS faster than the disk bandwidth of the DNs, it  saturates
the disk bandwidth and put the DNs unresponsive. The client only backs off by aborting / recovering
the pipeline, which leads to failed writes and unnecessary pipeline recovery.
> This jira proposes to add explicit congestion control mechanisms in the writing pipeline.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message