Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1383406852.1207701684463.JavaMail.jira@brutus>
Date: Tue, 8 Apr 2008 17:41:24 -0700 (PDT)
From: "Raghu Angadi (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-2910) Throttle IPC Client/Server during
 bursts of requests or server slowdown
In-Reply-To: <1655129481.1204139511113.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587022#action_12587022 ] 

Raghu Angadi commented on HADOOP-2910:
--------------------------------------

In addition to Doug's comment, though this patch indents accept all the incoming connections, looks like it is limited by the buffer size for the pipe (4k?).. once this buffer is full, Acceptor thread will spin, right?

Apart from accept policy, since we handle queue differently with the patch, should queue size be larger (at least in the order of number of client NameNode expects).. right now it is 100*handlers. I think 100 is too low. How would the system behave when 12k clients are expected and queue size is 4k?


> Throttle IPC Client/Server during bursts of requests or server slowdown
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-2910
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2910
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.16.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.18.0
>
>         Attachments: callQueue.patch, callQueue1.patch, callQueue2.patch, callQueue3.patch, throttleClient.patch
>
>
> I propose the following to avoid an IPC server being swarmed by too many requests and connections
> 1. Limit call queue length or limit the amount of memory used in the call queue. This can be done by including the size of a request in the header and storing unmarshaled requests in the call queue. 
> 2. If the call queue is full or queue buffer is full, stop reading requests from sockets. So requests stay at the server's system buffer or at the client side and thus eventually throttle the client. 
> 3. Limit the total number of connections. Do not accept new connections if the connection limit is exceeded. (Note: this solution is unfair to new connections.) 
> 4. If receive out of memory exception, close the current connection. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.