spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Rosen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-8319) Update several pieces of shuffle logic related to key orderings
Date Fri, 12 Jun 2015 00:51:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-8319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Josh Rosen updated SPARK-8319:
------------------------------
    Summary: Update several pieces of shuffle logic related to key orderings  (was: Enable
Tungsten shuffle manager for some shuffles that specify key orderings)

> Update several pieces of shuffle logic related to key orderings
> ---------------------------------------------------------------
>
>                 Key: SPARK-8319
>                 URL: https://issues.apache.org/jira/browse/SPARK-8319
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle, SQL
>            Reporter: Josh Rosen
>            Assignee: Josh Rosen
>
> The Tungsten ShuffleManager falls back to regular SortShuffleManager whenever the shuffle
dependency specifies a key ordering, but technically we only need to fall back when an aggregator
is also specified.  We should update the fallback logic to handle this case so that the Tungsten
optimizations can apply to more workloads.
> I also noticed that the SQL Exchange operator performs defensive copying of shuffle inputs
when a key ordering is specified, but this is unnecessary: the only shuffle manager that performs
sorting on the map side is SortShuffleManager, and it only performs sorting if an aggregator
is specified.  SQL never uses Spark's shuffle for performing aggregation, so this copying
is unnecessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message