hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Vary (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-17291) Set the number of executors based on config if client does not provide information
Date Mon, 14 Aug 2017 15:35:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125842#comment-16125842
] 

Peter Vary commented on HIVE-17291:
-----------------------------------

Thanks [~lirui], helpful as always!

What I do not get, why we do not see flakiness on TestMiniSparkOnYarnCliDriver tests, when
the we change a spark configuration value, and the spark session is invalidated. Until now
I thought, that in this case new containers are created by yarn, and we will have a race condition
when checking {{hiveSparkClient.getExecutorCount()}} again. But obviously this is not the
case, since I do not see any flakiness in the test results in HIVE-17292.

Thanks,
Peter

> Set the number of executors based on config if client does not provide information
> ----------------------------------------------------------------------------------
>
>                 Key: HIVE-17291
>                 URL: https://issues.apache.org/jira/browse/HIVE-17291
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>    Affects Versions: 3.0.0
>            Reporter: Peter Vary
>            Assignee: Peter Vary
>         Attachments: HIVE-17291.1.patch
>
>
> When calculating the memory and cores and the client does not provide information we
should try to use the one provided by default. This can happen on startup, when {{spark.dynamicAllocation.enabled}}
is not enabled



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message