spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Resolved] (SPARK-12737) Decrease the redundant activeIds sent to remote mirrors in "aggregateMessagesWithActiveSet"
Date Wed, 29 Jun 2016 13:36:31 GMT


Sean Owen resolved SPARK-12737.
    Resolution: Won't Fix

> Decrease the redundant activeIds sent to remote mirrors in "aggregateMessagesWithActiveSet"
> -------------------------------------------------------------------------------------------
>                 Key: SPARK-12737
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: GraphX
>    Affects Versions: 1.5.2
>            Reporter: qbwu
>              Labels: newbie
> Hi, I found that it is not necessary to send the activeIds to all the mirrors of some
master in the activeSetOpt which is passed to aggregateMessagesWithActiveSet. Because through
the passed EdgeDirection, we can infer what kind of mirrors (classified by their position)
will be checked isActive later, we can send the activeIds to the mirrors only at some position.
In some cases even no activeId needs to be sent.
> I have implemented it, and did some tests using PageRank and ConnectedComponent, the
shuffle size and the running time are decreased. But the number of runing turn is not changed.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message