hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Vary (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-17291) Set the number of executors based on config if client does not provide information
Date Mon, 14 Aug 2017 19:18:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126257#comment-16126257
] 

Peter Vary commented on HIVE-17291:
-----------------------------------

For testing this is exactly what I did in HIVE-17292.3.patch. I am not sure why we do not
have the flakiness in this case when the configuration is changed, and the spark session is
killed and restarted. Logic suggests, that in this case we will request new executors based
on the new settings, and we are in the middle of the query test file, so [~lirui]'s magic
does not applies here.
For example:
{code:title=spark_dynamic_partition_pruning_2.q}
EXPLAIN SELECT d1.label, count(*), sum(agg.amount) 
[..]

set hive.spark.dynamic.partition.pruning.max.data.size=1;   <-- I think new session is
started here

EXPLAIN SELECT d1.label, count(*), sum(agg.amount)
[..]
{code}

> Set the number of executors based on config if client does not provide information
> ----------------------------------------------------------------------------------
>
>                 Key: HIVE-17291
>                 URL: https://issues.apache.org/jira/browse/HIVE-17291
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>    Affects Versions: 3.0.0
>            Reporter: Peter Vary
>            Assignee: Peter Vary
>         Attachments: HIVE-17291.1.patch
>
>
> When calculating the memory and cores and the client does not provide information we
should try to use the one provided by default. This can happen on startup, when {{spark.dynamicAllocation.enabled}}
is not enabled



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message