hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chengxiang Li (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-7567) support automatic calculating reduce task number
Date Fri, 01 Aug 2014 12:31:38 GMT

     [ https://issues.apache.org/jira/browse/HIVE-7567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chengxiang Li updated HIVE-7567:
--------------------------------

    Attachment: HIVE-7567.1-spark.patch

support automatic adjusting reducer number same as MR, configure through 3 following parameters:
# In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
# In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
# In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>

Currently Tez specified optimization(committed with [HIVE-7158|https://issues.apache.org/jira/browse/HIVE-7158])
is not supported as it's based on Tez feature, refer to [dynamic graph reconfiguration|http://hortonworks.com/blog/apache-tez-dynamic-graph-reconfiguration/]
for details.

> support automatic calculating reduce task number
> ------------------------------------------------
>
>                 Key: HIVE-7567
>                 URL: https://issues.apache.org/jira/browse/HIVE-7567
>             Project: Hive
>          Issue Type: Task
>          Components: Spark
>            Reporter: Chengxiang Li
>            Assignee: Chengxiang Li
>         Attachments: HIVE-7567.1-spark.patch
>
>
> Hive have its own machenism to calculate reduce task number, we need to implement it
on spark job.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message