hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6762) exception while doing RPC I/O closes channel
Date Fri, 04 Jun 2010 02:15:55 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875427#action_12875427
] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

Hey Sam,

My cluster test case just spat out a nice warning and then deadlocked my DFS client:

10/06/03 18:48:54 WARN hdfs.DFSClient: Error Recovery for block blk_832465809601113490_3336818
in pipeline 192.168.42.40:11072, 192.168.42.41:11072, 192.168.42.43:11072: bad datanode 192.168.42.41:11072
Exception in thread "DataStreamer for file /user/todd/test-sync block blk_832465809601113490_3336818"
java.lang.reflect.UndeclaredThrowableException
        at $Proxy1.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)
        at org.apache.hadoop.hdfs.DFSClient.createClientDatanodeProtocolProxy(DFSClient.java:146)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2627)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2139)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2306)
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1238)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)
        at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:526)
        at org.apache.hadoop.ipc.Client.call(Client.java:765)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
        ... 8 more

I think we need to add another catch (InterruptedException) and rethrow as IOException around
the sendParam call in Client.call

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt,
hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(),
and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt
races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes
the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message