hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elliott Clark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9669) TcpPeerServer should respect ipc.server.listen.queue.size
Date Thu, 11 Feb 2016 19:04:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143279#comment-15143279
] 

Elliott Clark commented on HDFS-9669:
-------------------------------------

Nope I just missed it on a pull. Sorry about the confusion. Thanks for the many many backports
[~cmccabe]

> TcpPeerServer should respect ipc.server.listen.queue.size
> ---------------------------------------------------------
>
>                 Key: HDFS-9669
>                 URL: https://issues.apache.org/jira/browse/HDFS-9669
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.7.2
>            Reporter: Elliott Clark
>            Assignee: Elliott Clark
>             Fix For: 2.7.3
>
>         Attachments: HDFS-9669.0.patch, HDFS-9669.1.patch, HDFS-9669.1.patch
>
>
> On periods of high traffic we are seeing:
> {code}
> 16/01/19 23:40:40 WARN hdfs.DFSClient: Connection failure: Failed to connect to /10.138.178.47:50010
for file /MYPATH/MYFILE for block BP-1935559084-10.138.112.27-1449689748174:blk_1080898601_7375294:java.io.IOException:
Connection reset by peer
> java.io.IOException: Connection reset by peer
> 	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> 	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> 	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> 	at sun.nio.ch.IOUtil.write(IOUtil.java:65)
> 	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
> 	at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
> 	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> 	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
> 	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
> 	at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:109)
> 	at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
> {code}
> At the time that this happens there are way less xceivers than configured.
> On most JDK's this will make 50 the total backlog at any time. This effectively means
that any GC + Busy time willl result in tcp resets.
> http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/share/classes/java/net/ServerSocket.java#l370



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message