hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2789) Race condition in ipc.Server prevents responce being written back to client.
Date Thu, 07 Feb 2008 00:35:08 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12566415#action_12566415
] 

Raghu Angadi commented on HADOOP-2789:
--------------------------------------

Could you attach the log file when the test failed with and/or without the patch?

Most not the cause of test failure but related : doPurge() removes the responses that have
waited for too long. But it does not check if they are partially written. If we are not able
to send a partially write response, I think should close the connection (or update the recvTime
when any data is written).

> Race condition in ipc.Server prevents responce being written back to client.
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-2789
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2789
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.16.0
>            Reporter: Clint Morgan
>            Priority: Critical
>         Attachments: HADOOP-2789.patch
>
>
> I encountered a race condition in ipc.Server when writing the response
> back to the socket. Sometimes the write SelectKey is being canceled
> when it should not be, and thus the full response never gets
> written. This results in clients timing out on the socket while waiting for the response.
> I am attaching a unit test that demonstrates the problem. It follows
> closely the TestIPC test, however the socket output buffer is set
> smaller than the result being sent back, so that partial writes
> occur. I also put random sleep in the client to help provoke the race
> condition.
> On my machine this fails over half of the time.
> Looking at the code in ipc.Server.java. The problem is manifested in
> Responder.doAsyncWrite(). If I comment out the key.cancel() line, then
> everything works fine. 
> So we need to identify when to safely cancel the key.
> I tried the following:
> {noformat}
>     private void doAsyncWrite(SelectionKey key) throws IOException {
>       Call call = (Call)key.attachment();
>       if (call == null) {
>         return;
>       }
>       if (key.channel() != call.connection.channel) {
>         throw new IOException("doAsyncWrite: bad channel");
>       }
>       if (processResponse(call.connection.responseQueue)) {
>           synchronized(call.connection.responseQueue) {
>               if (call.connection.responseQueue.size() == 0) {
>                   LOG.info("Cancelling key for call "+call.toString()+ " key: "+ key.toString());
>                   key.cancel();          // remove item from selector.
>               } else {
>                   LOG.warn("NOT REALLY DONE: "+call.toString()+ " key: "+ key.toString());
>               }
>           }
>       }
>     }
> {noformat}
> And this does catch some of the cases (EG, the LOG.warn message gets hit), but i still
hit the race condition.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message