hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2910) Throttle IPC Client/Server during bursts of requests or server slowdown
Date Tue, 18 Mar 2008 18:09:28 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579971#action_12579971
] 

Raghu Angadi commented on HADOOP-2910:
--------------------------------------

> Ok, to avoid busy waiting, the listener checks if the call queue is full or not before
it calls select. It waits if the queue is full. How does this sound?

+1. This is much simpler than fiddling with the keys. 

One minor improvement we could have now or in future is to make the selector thread wait for
a low water mark (say 90% of max call queue), when it decides to wait for call queue to be
drained. this would reduce number of selector loops when the queue is hovering at max.

> I think it is ok to accept new connections when the queue is full. 

Not selecting as you suggested will stop accepting new connections, which I think is good.
It will fail clients only in extreme cases (ie queue stays full for many minutes).. in such
cases, I guess 

> Throttle IPC Client/Server during bursts of requests or server slowdown
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-2910
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2910
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.16.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.17.0
>
>         Attachments: callQueue.patch
>
>
> I propose the following to avoid an IPC server being swarmed by too many requests and
connections
> 1. Limit call queue length or limit the amount of memory used in the call queue. This
can be done by including the size of a request in the header and storing unmarshaled requests
in the call queue. 
> 2. If the call queue is full or queue buffer is full, stop reading requests from sockets.
So requests stay at the server's system buffer or at the client side and thus eventually throttle
the client. 
> 3. Limit the total number of connections. Do not accept new connections if the connection
limit is exceeded. (Note: this solution is unfair to new connections.) 
> 4. If receive out of memory exception, close the current connection. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message