hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8829) Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning
Date Mon, 01 Feb 2016 21:46:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127097#comment-15127097
] 

Colin Patrick McCabe commented on HDFS-8829:
--------------------------------------------

bq. There is one corner cases: Old linux kernels do not support auto tuning. I am not sure
what would happen when setting these values to 0.

If we don't explicitly set the send and receive buffer sizes, and there is no auto-tuning,
presumably we will receive the defaults.  I think in practice this is not a very interesting
or important case.  Admins know the performance characteristics of old clusters pretty well
by now, and when they upgrade their clusters, they'll be on newer kernels which will have
the auto-tuning feature.

> Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow
configuring auto-tuning
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-8829
>                 URL: https://issues.apache.org/jira/browse/HDFS-8829
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.3.0, 2.6.0
>            Reporter: He Tianyi
>            Assignee: He Tianyi
>             Fix For: 2.8.0
>
>         Attachments: HDFS-8829.0001.patch, HDFS-8829.0002.patch, HDFS-8829.0003.patch,
HDFS-8829.0004.patch, HDFS-8829.0005.patch, HDFS-8829.0006.patch
>
>
> {code:java}
>   private void initDataXceiver(Configuration conf) throws IOException {
>     // find free port or use privileged port provided
>     TcpPeerServer tcpPeerServer;
>     if (secureResources != null) {
>       tcpPeerServer = new TcpPeerServer(secureResources);
>     } else {
>       tcpPeerServer = new TcpPeerServer(dnConf.socketWriteTimeout,
>           DataNode.getStreamingAddr(conf));
>     }
>     tcpPeerServer.setReceiveBufferSize(HdfsConstants.DEFAULT_DATA_SOCKET_SIZE);
> {code}
> The last line sets SO_RCVBUF explicitly, thus disabling tcp auto-tuning on some system.
> Shall we make this behavior configurable?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message