spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Iurii Antykhovych (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-21491) Performance enhancement: eliminate creation of intermediate collections
Date Fri, 21 Jul 2017 00:00:46 GMT

    [ https://issues.apache.org/jira/browse/SPARK-21491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095587#comment-16095587
] 

Iurii Antykhovych commented on SPARK-21491:
-------------------------------------------

I searched for all such places in the whole GraphX module.
These are only three occurrences of such tuples-to-map conversion I could boost without major
refactoring.
Let GraphX be the pilot module of such an optimization)

The only change on a hot path of execution is the code is the one on `graphx.lib.ShortestPaths`
class.
The rest is executed once per run, not 'hot' enough. 
Shall I revert it (LabelPropagation.scala, PageRank.scala)?



> Performance enhancement: eliminate creation of intermediate collections
> -----------------------------------------------------------------------
>
>                 Key: SPARK-21491
>                 URL: https://issues.apache.org/jira/browse/SPARK-21491
>             Project: Spark
>          Issue Type: Improvement
>          Components: GraphX
>    Affects Versions: 2.2.0
>            Reporter: Iurii Antykhovych
>            Priority: Trivial
>
> Simple performance optimization in a few places of GraphX:
> {{Traversable.toMap}} can be replaced with {{collection.breakout}}.
> This would eliminate creation of an intermediate collection of tuples, see
> [Stack Overflow article|https://stackoverflow.com/questions/1715681/scala-2-8-breakout]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message