hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12909) Change ipc.Client to support asynchronous calls
Date Mon, 21 Mar 2016 16:55:25 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204595#comment-15204595

Sanjay Radia commented on HADOOP-12909:

I haven't had a chance to look at the patch or review all the comments, but wanted to bring
attention to one issue wrt async rpc that is well known by implementors and practitioners
of message passing & rpc systems (excuse me if this has already been covered): 
* One needs to watch out for buffer management. ie. aync rpc/message passing has the potential
to use up memory for buffering the messages. This is prevented in Sync rpc systems: 
** the sender (client) blocks and cannot flood the receiver unless it uses threads
** the receiver (server) is guaranteed that the sender (ie client) is waiting to receive and
if it has died then the reply can be discarded.

With asyn rpc , my suggestion is to consider something along the following lines:
*  the client needs to allocate some buffer (or space for it)  where replies are stored. On
each async rpc call, it passes a ref to this buffer for storing replies. If the client does
not pick up the replies fast enough then his next async call using that buffer space will
* Note this makes the clients code tricky in what to do if it is blocked since one must ensure
that a deadlock or starvation  does not happen (but async messaging has always been tricky
which is why cs community went with sync rpc). Note this problem does not arise on server
side async-rpc since the client is blocked waiting for reply (unless the client also did async
call but in that case its buffer, as per my suggestion,  must be there to store the reply).

> Change ipc.Client to support asynchronous calls
> -----------------------------------------------
>                 Key: HADOOP-12909
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12909
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Xiaobing Zhou
>         Attachments: HADOOP-12909-HDFS-9924.000.patch, HADOOP-12909-HDFS-9924.001.patch,
HADOOP-12909-HDFS-9924.002.patch, HADOOP-12909-HDFS-9924.003.patch
> In ipc.Client, the underlying mechanism is already supporting asynchronous calls -- the
calls shares a connection, the call requests are sent using a thread pool and the responses
can be out of order.  Indeed, synchronous call is implemented by invoking wait() in the caller
thread in order to wait for the server response.
> In this JIRA, we change ipc.Client to support asynchronous mode.  In asynchronous mode,
it return once the request has been sent out but not wait for the response from the server.

This message was sent by Atlassian JIRA

View raw message