hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@yahoo-inc.com>
Subject blocking read on a socket blocks write too?
Date Tue, 05 Feb 2008 16:41:39 GMT
As part of HADOOP-2346, I changed how default socketFactory creates its 
socket : from 'new Socket()' to 'SocketChannel.open().socket()' since 
DFSClient uses socket factory. As a side effect, ipc.Client ends up 
using this socket creation.

ipc.Client's internal response receiver is blocked on 
socket.getInputStream().read() as expected. But other threads can not 
send RPCs since their write() is blocked on a socket imlementation 
related lock. This lock is held by the reader. Thats pretty weird. How 
can it not allow read and write at the same time?

Raghu.

Related stack traces (from a Junit test):

read : locked 0xd32b6bf0
=============================
"IPC Client connection to localhost/127.0.0.1:55567" daemon prio=10 
tid=0xc02dfc00 nid=0x2c41 runnable [0xbf836000..0xbf836fb0]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:184)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0xd32c30c8> (a sun.nio.ch.Util$1)
	- locked <0xd32c30b8> (a java.util.Collections$UnmodifiableSet)
	- locked <0xd32c2ed0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:193)
	- locked <0xd32b6bf0> (a java.lang.Object)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
	- locked <0xd32b6f50> (a sun.nio.ch.SocketAdaptor$SocketInputStream)
	at java.io.FilterInputStream.read(FilterInputStream.java:116)
	at org.apache.hadoop.ipc.Client$Connection$1.read(Client.java:190)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
	- locked <0xd32b6f10> (a java.io.BufferedInputStream)
	at java.io.DataInputStream.readInt(DataInputStream.java:370)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:276)

write : trying to lock 0xd32b6bf0
==================================
"main" prio=10 tid=0x0805b000 nid=0x2b85 waiting for monitor entry 
[0xd7e66000..0xd7e67288]
    java.lang.Thread.State: BLOCKED (on object monitor)
	at java.nio.channels.Channels.write(Channels.java:58)
	- waiting to lock <0xd32b6bf0> (a java.lang.Object)
	at java.nio.channels.Channels.access$000(Channels.java:47)
	at java.nio.channels.Channels$1.write(Channels.java:134)
	- locked <0xd32b90f8> (a java.nio.channels.Channels$1)
	at org.apache.hadoop.ipc.Client$Connection$2.write(Client.java:201)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
	- locked <0xd32b90c0> (a java.io.BufferedOutputStream)
	at java.io.DataOutputStream.flush(DataOutputStream.java:106)
	at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:346)
	- locked <0xd32b90a8> (a java.io.DataOutputStream)
	at org.apache.hadoop.ipc.Client.call(Client.java:504)
	- locked <0xd32bb3b0> (a org.apache.hadoop.ipc.Client$Call)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
	at org.apache.hadoop.dfs.$Proxy1.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:291)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:278)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:315)
	at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:260)
	at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:211)
	at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:166)
	at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:2532)
	at org.apache.hadoop.dfs.DataNode.run(DataNode.java:2476)
	at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:2497)
	at 
org.apache.hadoop.dfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:298)


Mime
View raw message