hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8649) Increase level of parallelism in reduce phase [Spark Branch]
Date Thu, 06 Nov 2014 19:23:34 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200711#comment-14200711
] 

Xuefu Zhang commented on HIVE-8649:
-----------------------------------

Some comments on review board.

> Increase level of parallelism in reduce phase [Spark Branch]
> ------------------------------------------------------------
>
>                 Key: HIVE-8649
>                 URL: https://issues.apache.org/jira/browse/HIVE-8649
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Brock Noland
>            Assignee: Jimmy Xiang
>             Fix For: spark-branch
>
>         Attachments: HIVE-8649.1-spark.patch
>
>
> We calculate the number of reducers based on the same code for MapReduce. However, reducers
are vastly cheaper in Spark and it's generally recommended we have many more reducers than
in MR.
> Sandy Ryza who works on Spark has some ideas about a heuristic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message