Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <30438707.1202771588829.JavaMail.jira@brutus>
Date: Mon, 11 Feb 2008 15:13:08 -0800 (PST)
From: "Clint Morgan (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-2789) Race condition in ipc.Server prevents
 responce being written back to client.
In-Reply-To: <3571230.1202335290995.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HADOOP-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Clint Morgan updated HADOOP-2789:
---------------------------------

    Attachment: failure-after-patch.log.gz

> Race condition in ipc.Server prevents responce being written back to client.
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-2789
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2789
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.16.0
>            Reporter: Clint Morgan
>            Assignee: Raghu Angadi
>            Priority: Critical
>             Fix For: 0.16.1
>
>         Attachments: failure-after-patch.log.gz, failure-with-patch.log, failure.log, HADOOP-2789-Test.patch, HADOOP-2789.patch, HADOOP-2789.patch, HADOOP-2789.patch, HADOOP-2789.patch, HADOOP-2789.patch, HADOOP-2789.patch, success.log
>
>
> I encountered a race condition in ipc.Server when writing the response
> back to the socket. Sometimes the write SelectKey is being canceled
> when it should not be, and thus the full response never gets
> written. This results in clients timing out on the socket while waiting for the response.
> I am attaching a unit test that demonstrates the problem. It follows
> closely the TestIPC test, however the socket output buffer is set
> smaller than the result being sent back, so that partial writes
> occur. I also put random sleep in the client to help provoke the race
> condition.
> On my machine this fails over half of the time.
> Looking at the code in ipc.Server.java. The problem is manifested in
> Responder.doAsyncWrite(). If I comment out the key.cancel() line, then
> everything works fine. 
> So we need to identify when to safely cancel the key.
> I tried the following:
> {noformat}
>     private void doAsyncWrite(SelectionKey key) throws IOException {
>       Call call = (Call)key.attachment();
>       if (call == null) {
>         return;
>       }
>       if (key.channel() != call.connection.channel) {
>         throw new IOException("doAsyncWrite: bad channel");
>       }
>       if (processResponse(call.connection.responseQueue)) {
>           synchronized(call.connection.responseQueue) {
>               if (call.connection.responseQueue.size() == 0) {
>                   LOG.info("Cancelling key for call "+call.toString()+ " key: "+ key.toString());
>                   key.cancel();          // remove item from selector.
>               } else {
>                   LOG.warn("NOT REALLY DONE: "+call.toString()+ " key: "+ key.toString());
>               }
>           }
>       }
>     }
> {noformat}
> And this does catch some of the cases (EG, the LOG.warn message gets hit), but i still hit the race condition.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.