spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (Jira)" <>
Subject [jira] [Updated] (SPARK-30432) reduce degree recomputation in StronglyConnectedComponents
Date Mon, 13 Jan 2020 05:28:00 GMT


Dongjoon Hyun updated SPARK-30432:
    Target Version/s:   (was: 2.4.5, 3.0.0)

> reduce degree recomputation in StronglyConnectedComponents
> ----------------------------------------------------------
>                 Key: SPARK-30432
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: GraphX
>    Affects Versions: 3.0.0
>            Reporter: li xiaosen
>            Priority: Major
> So the computation happens every time in the do-while loop, the first time the outer
while loop executes. although just once per do-while loop after, it seems, but It does reduce
a lot of recomputation;because every time it jump out of the do-while loop,there are
no vertices have only out-degree or in-degree,so it's no need to recompute degree to tag
the vertices true.
> I have done a small code proposal, because there is a problem when the pregel executions
have done,  the degree no need to be recomputed.
> for example,the Email-EuAll  data set:[]
> do-while loop execute 10 times,and the reduce logic happend 2 times;so it would be
helpful when computing StronglyConnectedComponents to reduce degree computation.
> I created a branch in my fork: []
> I hope you can consider this small code proposal.
> Thank you very much,
> Best regards,
> xs-li

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message