hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-7270) Add congestion signaling capability to DataNode write protocol
Date Fri, 06 Feb 2015 16:31:37 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309397#comment-14309397
] 

Daryn Sharp edited comment on HDFS-7270 at 2/6/15 4:31 PM:
-----------------------------------------------------------

> Both enum and a uint32 are encoded as a varint32 over the wire. ... Can you clarify what
compatibility means in your mind? 

Yes, I understand how/why that technically works over the wire.  Clever, but bad because modifying
an existing protobuf tag is akin to changing a public API's method return signature.  We don't
do that, unless perhaps there's an extraordinary circumstance where the API is insecure or
completely broken.

This isn't an extraordinary case.  The feature provides a client advisory which newer clients
aren't obligated honor.  Older clients are broken just to save 1 byte per datanode in the
pipeline ack due to the overloading of the Status field to utilize more bits.

Simply adding a new optional tag, whose bits can be used to encode more than ECN in the future,
is a completely compatible change.


was (Author: daryn):
> Both enum and a uint32 are encoded as a varint32 over the wire. ... Can you clarify what
compatibility means in your mind? 

Yes, I understand how/why that technically works over the wire.  Clever, but bad because modifying
an existing protobuf tag to changing a public API's method return signature.  We don't do
that, unless perhaps there's an extraordinary circumstance where the API is insecure or completely
broken.

This isn't an extraordinary case.  The feature provides a client advisory which newer clients
aren't obligated honor.  Older clients are broken just to save 1 byte per datanode in the
pipeline ack due to the overloading of the Status field to utilize more bits.

Simply adding a new optional tag, whose bits can be used to encode more than ECN in the future,
is a completely compatible change.

> Add congestion signaling capability to DataNode write protocol
> --------------------------------------------------------------
>
>                 Key: HDFS-7270
>                 URL: https://issues.apache.org/jira/browse/HDFS-7270
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>            Reporter: Haohui Mai
>            Assignee: Haohui Mai
>         Attachments: HDFS-7270.000.patch, HDFS-7270.001.patch, HDFS-7270.002.patch, HDFS-7270.003.patch,
HDFS-7270.004.patch
>
>
> When a client writes to HDFS faster than the disk bandwidth of the DNs, it  saturates
the disk bandwidth and put the DNs unresponsive. The client only backs off by aborting / recovering
the pipeline, which leads to failed writes and unnecessary pipeline recovery.
> This jira proposes to add explicit congestion control mechanisms in the writing pipeline.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message