hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Jungblut (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-593) Improve RPC scalability
Date Thu, 09 Aug 2012 08:00:30 GMT

    [ https://issues.apache.org/jira/browse/HAMA-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431653#comment-13431653

Thomas Jungblut commented on HAMA-593:

In both of the managers (Hadoop|Avro) the RPC connections to the other peers are cached in
a map called "peers".

This will always leave the connection open to every other task, imagine you have 1k tasks,
each task will hold 1k connections to each other (999 outband, 1 local).

1. The caching must be removed and the connections must be closed when the messages were send.

Then, there is a problem when all 1k peers would attempt to send to a single peer (let's say
a master task in a graph algorithm that aggregates). In this case the peer will start 1k-threads
which is using enourmous amount of memory. I wouldn't mind if this can be done smarter, but
I have no solution at hand currently.

Would be cool if 1. could be resolved :)
> Improve RPC scalability
> -----------------------
>                 Key: HAMA-593
>                 URL: https://issues.apache.org/jira/browse/HAMA-593
>             Project: Hama
>          Issue Type: Sub-task
>          Components: bsp core, messaging
>    Affects Versions: 0.5.0
>            Reporter: Thomas Jungblut
> To improve scalability we can start a RPC connection after another instead of keeping
all possible N connections open the whole time.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message