hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7270) Add congestion signaling capability to DataNode write protocol
Date Thu, 05 Feb 2015 23:42:36 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308262#comment-14308262
] 

Daryn Sharp commented on HDFS-7270:
-----------------------------------

I've looked at the patch again.  The problem is changing the existing protobuf Status enum
record tag from an enum to a uint32.  That pretty much violates the compatibility promise
that protobufs are supposed to provide.

The reason the tag was changed was to allow masking the ECN bit atop the status value.  If
it's off, 0 is masked onto the status which is why it "works".

The compatible solution is do what we always do:  Add a new optional protobuf tag for ECN.
 It's a dangerous precedent to allow breaking compatibility just to save nominally 3 bytes
across the wire.

> Add congestion signaling capability to DataNode write protocol
> --------------------------------------------------------------
>
>                 Key: HDFS-7270
>                 URL: https://issues.apache.org/jira/browse/HDFS-7270
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>            Reporter: Haohui Mai
>            Assignee: Haohui Mai
>         Attachments: HDFS-7270.000.patch, HDFS-7270.001.patch, HDFS-7270.002.patch, HDFS-7270.003.patch,
HDFS-7270.004.patch
>
>
> When a client writes to HDFS faster than the disk bandwidth of the DNs, it  saturates
the disk bandwidth and put the DNs unresponsive. The client only backs off by aborting / recovering
the pipeline, which leads to failed writes and unnecessary pipeline recovery.
> This jira proposes to add explicit congestion control mechanisms in the writing pipeline.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message