giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jakob Homan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-256) Partitioning outgoing graph data during INPUT_SUPERSTEP by # of vertices results in wide variance in RPC message sizes
Date Tue, 07 Aug 2012 23:37:10 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430725#comment-13430725
] 

Jakob Homan commented on GIRAPH-256:
------------------------------------

Looking good.  Can you move the new constants to the classes that use them, rather than GiraphJob?
Also, there are still some spurious white space changes to be removed.  Thanks.
                
> Partitioning outgoing graph data during INPUT_SUPERSTEP by # of vertices results in wide
variance in RPC message sizes
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: GIRAPH-256
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-256
>             Project: Giraph
>          Issue Type: Improvement
>          Components: bsp, graph
>    Affects Versions: 0.2.0
>            Reporter: Eli Reisman
>            Assignee: Eli Reisman
>              Labels: patch
>             Fix For: 0.2.0
>
>         Attachments: GIRAPH-256-1.patch, GIRAPH-256-2.patch, GIRAPH-256-3.patch, GIRAPH-256-4.patch,
GIRAPH-256-5.patch, GIRAPH-256-6.patch
>
>
> This relates to GIRAPH-247. The unfortunately named "MAX_VERTICES_PER_PARTITION" fooled
me into thinking this value was regulating the size of initial Partition objects as they were
composed during INPUT_SUPERSTEP from InputSplits each worker reads.
> In fact this configuration option only regulates the size of the outgoing RPC messages,
stored locally in Partition objects but decomposed into Collections of BasicVertex for transfer
to their eventual homes on another (or this) worker. There they are combined into the actual
Partitions they will exist in for the job run.
> By partitioning these outgoing messages by # of vertices, metrics load tests have shown
the size of the average message is not well regulated and can create overloads on either side
of these transfers. This is important because:
> 1. Throughput and memory are at a premium during INPUT_SUPERSTEP.
> 2. Only one crashed worker in a Giraph job causes cascading job failure, even in an otherwise
healthy workflow.
> This JIRA renames the offending variables/config options and further regulates outgoing
graph data in INPUT_SUPERSTEP by the # of edges and THEN the # of vertices in a candidate
for transfer. This much more effectively regulates message size for typical social graph data
and has been show in testing to greatly improve the amount of load-in data Giraph can handle
without failure given fixed memory and worker limits.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message