hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck
Date Mon, 19 Apr 2010 06:11:50 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858390#action_12858390
] 

dhruba borthakur commented on HADOOP-6713:
------------------------------------------

My proposal is that the Server.Listener.run() thread only executes doAccept(). Also there
are a bunch of open Selector objects (instead of just one Selector object in the current code).
 Each of these Selector objects are handled by its own thread.

The Server.Listener.run() invoke doAccept() that inserts a SocketChannel into a randomly selected
Selector object. The thread that is monitoring this Selector object will then invoke doRead()
and process the RPC.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from
the socket and puts them into a call queue. This means that this single thread can pull RPC
requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck
in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message