hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2910) Throttle IPC Client/Server during bursts of requests or server slowdown
Date Wed, 12 Mar 2008 23:28:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12578073#action_12578073

Hairong Kuang commented on HADOOP-2910:

> Limit the total number of connections.
Most operation systems provide some parameters to limit the total number of connections on
the machine. It seems that it is unnecessary for the IPC server to concern about it. So I
drop this requirement in this jira.

Regarding to limiting the resource consumed by the incoming requests, we could either
Solution 1. Limit call queue length.  The ipc server stops reading from any connection if
the call queue is full.
Solution 2. Limit the amount of memory used in the call queue. This requires the following
    * Call queue contains unmarshaled RPC requests;
    * Requests get serialized after being taken out of the call queue and before being served;
    * Keep count of the total size of the partial and complete unmarshaled requests;
    * Stop reading from a connection if the incoming request size + total size > max buffer

Solution 1 is simple. Solution 2 provides more accurate memory management and parallel request
serialization. Potentially we could do a buffer management to eliminate the frequent buffer
allocation & deallocation discussed in HADOOP-2975. But solution 2 is more complicated
and not favorable to large messages like block reports.

Please comment! Thank you.

> Throttle IPC Client/Server during bursts of requests or server slowdown
> -----------------------------------------------------------------------
>                 Key: HADOOP-2910
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2910
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.16.0
>            Reporter: Hairong Kuang
>             Fix For: 0.17.0
> I propose the following to avoid an IPC server being swarmed by too many requests and
> 1. Limit call queue length or limit the amount of memory used in the call queue. This
can be done by including the size of a request in the header and storing unmarshaled requests
in the call queue. 
> 2. If the call queue is full or queue buffer is full, stop reading requests from sockets.
So requests stay at the server's system buffer or at the client side and thus eventually throttle
the client. 
> 3. Limit the total number of connections. Do not accept new connections if the connection
limit is exceeded. (Note: this solution is unfair to new connections.) 
> 4. If receive out of memory exception, close the current connection. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message