hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gary Helmling (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-9700) DFSClient and DFSOutputStream do not respect TCP_NODELAY config in two spots
Date Wed, 03 Feb 2016 02:48:39 GMT

     [ https://issues.apache.org/jira/browse/HDFS-9700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Gary Helmling updated HDFS-9700:
    Attachment: HDFS-9700_branch-2.7-v2.patch

Here are updated patches for trunk and branch-2.7.  These change the setting to use a new
config key: "dfs.client.socket.tcpnodelay".  This defaults to true, since HADOOP-8069 has
already update the default for ipc.(client|server).tcpnodelay to true.

I'll see if I can do some basic benchmarks with TestDFSIO with and without the setting.  I
don't expect this to make things worse, but neither do I expect much improvement in a throughput
test like TestDFSIO.

Where we do see a difference from this is in the P95 write latencies for our HBase clusters
with HDFS security enabled.

> DFSClient and DFSOutputStream do not respect TCP_NODELAY config in two spots
> ----------------------------------------------------------------------------
>                 Key: HDFS-9700
>                 URL: https://issues.apache.org/jira/browse/HDFS-9700
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client
>    Affects Versions: 2.7.1, 2.6.3
>            Reporter: Gary Helmling
>            Assignee: Gary Helmling
>         Attachments: HDFS-9700-v1.patch, HDFS-9700-v2.patch, HDFS-9700_branch-2.7-v2.patch,
> In {{DFSClient.connectToDN()}} and {{DFSOutputStream.createSocketForPipeline()}}, we
never call {{setTcpNoDelay()}} on the constructed socket before sending.  In both cases, we
should respect the value of ipc.client.tcpnodelay in the configuration.
> While this applies whether security is enabled or not, it seems to have a bigger impact
on latency when security is enabled.

This message was sent by Atlassian JIRA

View raw message