hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmytro Molkov (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-6713) The RPC server Listener thread is a scalability bottleneck
Date Mon, 26 Apr 2010 21:25:34 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dmytro Molkov updated HADOOP-6713:
----------------------------------

    Attachment: HADOOP-6713.2.patch

modifying the patch to address Hairong's comments:

1. Removed the System.out
2. Removed doRead from listener#run
3. Changed the code to accept all the connections in the queue

4. I wrote a comment on startAdd method to explain the synchronization a little. The reason
I went with this decision is to keep the semantics of doAccept. After the iteration with certain
channel that channel is registered with the selector and is ready to be read. Otherwise it
possibly could affect other parts of the code, when the connection is accepted (doAccept iteration
for the channel is over) but connectionList is not modified yet. And the connection is only
created when the channel is registered with selector.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from
the socket and puts them into a call queue. This means that this single thread can pull RPC
requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck
in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message