hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "He Tianyi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10326) Disable setting tcp socket send/receive buffers for write pipelines
Date Tue, 26 Apr 2016 23:29:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259173#comment-15259173
] 

He Tianyi commented on HDFS-10326:
----------------------------------

[~cmccabe] [~mingma]
https://www.kernel.org/doc/ols/2009/ols2009-pages-169-184.pdf
This document suggests auto tuning is not introduced in Linux 2.4 before 2.4.27 or Linux 2.6
before 2.6.7.
That's very old.

So maybe it's appropriate to enable auto tuning by default.

> Disable setting tcp socket send/receive buffers for write pipelines
> -------------------------------------------------------------------
>
>                 Key: HDFS-10326
>                 URL: https://issues.apache.org/jira/browse/HDFS-10326
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode, hdfs
>    Affects Versions: 2.6.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>
> The DataStreamer and the Datanode use a hardcoded DEFAULT_DATA_SOCKET_SIZE=128K for the
send and receive buffers of a write pipeline.  Explicitly setting tcp buffer sizes disables
tcp stack auto-tuning.  
> The hardcoded value will saturate a 1Gb with 1ms RTT.  105Mbs at 10ms.  Paltry 11Mbs
over a 100ms long haul.  10Gb networks are underutilized.
> There should either be a configuration to completely disable setting the buffers, or
the the setReceiveBuffer and setSendBuffer should be removed entirely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message