giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-328) Outgoing messages from current superstep should be grouped at the sender by owning worker, not by partition
Date Wed, 03 Oct 2012 03:08:07 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468286#comment-13468286
] 

Hudson commented on GIRAPH-328:
-------------------------------

Integrated in Giraph-trunk-Commit #217 (See [https://builds.apache.org/job/Giraph-trunk-Commit/217/])
    GIRAPH-328: Outgoing messages from current superstep should be grouped
at the sender by owning worker, not by partition. (Eli Reisman via
aching) (Revision 1393266)

     Result = FAILURE
aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1393266
Files : 
* /giraph/trunk/CHANGELOG
* /giraph/trunk/src/main/java/org/apache/giraph/comm/SendMessageCache.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/RequestType.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/SendPartitionCurrentMessagesRequest.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/SendPartitionMessagesRequest.java
* /giraph/trunk/src/main/java/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java
* /giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceMaster.java
* /giraph/trunk/src/main/java/org/apache/giraph/graph/WorkerInfo.java
* /giraph/trunk/src/test/java/org/apache/giraph/comm/RequestFailureTest.java
* /giraph/trunk/src/test/java/org/apache/giraph/comm/RequestTest.java

                
> Outgoing messages from current superstep should be grouped at the sender by owning worker,
not by partition
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: GIRAPH-328
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-328
>             Project: Giraph
>          Issue Type: Improvement
>          Components: bsp, graph
>    Affects Versions: 0.2.0
>            Reporter: Eli Reisman
>            Assignee: Eli Reisman
>            Priority: Minor
>             Fix For: 0.2.0
>
>         Attachments: GIRAPH-328-1.patch, GIRAPH-328-2.patch, GIRAPH-328-3.patch, GIRAPH-328-4.patch,
GIRAPH-328-5.patch, GIRAPH-328-6.patch, GIRAPH-328-7.patch, GIRAPH-328.8.patch
>
>
> Currently, outgoing messages created by the Vertex#compute() cycle on each worker are
stored and grouped by the partitionId on the destination worker to which the messages belong.
This results in messages being duplicated on the wire per partition on a given receiving worker
that has delivery vertices for those messages.
> By partitioning the outgoing, current-superstep messages by destination worker, we can
split them into partitions at insertion into a MessageStore on the destination worker. What
we trade in come compute time while inserting at the receiver side, we gain in fine grained
control over the real number of messages each worker caches outbound for any given worker
before flushing, and how those flush messages are aggregated for delivery as well. Potentially,
it allows for a great reduction in duplicate messages sent in situations like Vertex#sendMessageToAllEdges()
-- see GIRAPH-322, GIRAPH-314. You get the idea.
> This might be a poor idea, and it can certainly use some additional refinement, but it
passes mvn verify and may even run ;) It interoperates with the disk spill code, but not as
well as it could. Consider this a request for comment on the idea (and the approach) rather
than a finished product.
> Comments/ideas/help welcome! Thanks

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message