incubator-giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-128) RPC port from BasicRPCCommunications should be only a starting port, and retried
Date Sat, 28 Jan 2012 01:51:10 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195330#comment-13195330
] 

jiraposter@reviews.apache.org commented on GIRAPH-128:
------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3596/
-----------------------------------------------------------

(Updated 2012-01-28 01:15:26.114994)


Review request for giraph.


Changes
-------

Removed whitspace changes for MinimumIntCombiner.java and SimpleSumCombiner.java and made
GiraphJob.MAX_RPC_PORT_BIND_ATTEMPTS configurable, but default to 20.


Summary
-------

Simple handling of port collisions on the same machine while preserving debugability from
the port number alone.  Round up the max number of workers to the next power of 10 and use
it as a constant to increase the port number with.

Added a unit test to ensure it is working correctly.

Fixed 2 minor warnings on
src/main/java/org/apache/giraph/examples/MinimumIntCombiner.java
src/main/java/org/apache/giraph/examples/SimpleSumCombiner.java

of removing 'import java.util.List'.


This addresses bug GIRAPH-128.
    https://issues.apache.org/jira/browse/GIRAPH-128


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
1236935 
  http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java
1236935 
  http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/comm/RPCCommunicationsTest.java
PRE-CREATION 

Diff: https://reviews.apache.org/r/3596/diff


Testing
-------

Passed local and MR unittests.


Thanks,

Avery


                
> RPC port from BasicRPCCommunications should be only a starting port, and retried
> --------------------------------------------------------------------------------
>
>                 Key: GIRAPH-128
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-128
>             Project: Giraph
>          Issue Type: Improvement
>    Affects Versions: 0.1.0
>            Reporter: Avery Ching
>            Assignee: Avery Ching
>         Attachments: GIRAPH-128.2.patch
>
>
> Currently Giraph uses a basic port + the task partition to get the RPC port.  This doesn't
work well for when there are multiple Giraph jobs running simultaneously in the same Hadoop
cluster (port conflict).  At the same time, it is nice to use this simple algorithm because
it makes it very easy to debug problems (you can find the troublesome mapper from the RPC
port name).  I will be proposing a simple scheme to retry with another port.  I will round
the total number of mappers up to the nearest power of 10 (let's that that number Z).  Then
I will increment the port number by Z, retrying up to 20 tries.  If you have enough ports,
this scheme would guarantee that up to 20 mappers / node would be supported.  It should be
sufficient for most clusters.  At the same time, we still maintain the easy debugging method
since you it's still easy to figure out the mapper partition from the port (port % Z = map
partition). 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message