incubator-giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
Date Wed, 14 Dec 2011 19:51:32 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169630#comment-13169630
] 

jiraposter@reviews.apache.org commented on GIRAPH-57:
-----------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3201/
-----------------------------------------------------------

(Updated 2011-12-14 19:50:29.358317)


Review request for giraph.


Changes
-------

Forgot a few important files (VertexIdMessages.java and VertexIdMessagesList.java)


Summary
-------

Added new putVertexIdMessagesList RPC and supporting classes (VertexIdMessages and VertexIdMessagesList)
to reduce the total number of RPCs during a flush.  This improves the number of RPC / sec
and overall I/O bandwidth.  The amount of batching is done by the total number of messages
and is configurable at runtime (default of 5000, weighted toward helping smaller messages).
 I have noted some performance results in https://issues.apache.org/jira/browse/GIRAPH-57
(between 25 - 1075 percent improvements).

Also, while tinkering with BasicRPCCommunications, notices inconsistent spaces between 'synchronized'
and '('.  Removed spaces and standardized in the CODE_CONVENTIONS.


This addresses bug GIRAPH-57.
    https://issues.apache.org/jira/browse/GIRAPH-57


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/incubator/giraph/trunk/CODE_CONVENTIONS 1213849 
  http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ArrayListWritable.java
1213849 
  http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
1214406 
  http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java
1213849 
  http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java
1213849 

Diff: https://reviews.apache.org/r/3201/diff


Testing
-------

Passed local and Hadoop unittests.  Used the RandomMessageBenchmark on a small cluster.


Thanks,

Avery


                
> Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
> ----------------------------------------------------------------------------
>
>                 Key: GIRAPH-57
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-57
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Jakob Homan
>            Assignee: Avery Ching
>         Attachments: GIRAPH-57.diff
>
>
> Right now messages are sent to a vertex one at a time.  It would be good to have a putMsgs
call that could send messages to multiple vertices (all hosted on the same worker).  We'd
save a huge number of individual RPC calls at the expense of having smaller calls with larger
payloads.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message