hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-17270) Qtest results show wrong number of executors
Date Thu, 10 Aug 2017 03:59:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-17270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121020#comment-16121020
] 

Rui Li commented on HIVE-17270:
-------------------------------

Hi [~pvary], I don't understand why 1 NM means we can only have 1 executor. IIUC, NM can allocate
executor/container if there's enough memory. Each executor requires 512MB mem, and the AM
take another 512MB. So we use 1536MB in total. We start the NM with [2048MB total mem|https://github.com/apache/hive/blob/master/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java#L493],
which should be able to satisfy all the requests.

> Qtest results show wrong number of executors
> --------------------------------------------
>
>                 Key: HIVE-17270
>                 URL: https://issues.apache.org/jira/browse/HIVE-17270
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: 3.0.0
>            Reporter: Peter Vary
>            Assignee: Peter Vary
>
> The hive-site.xml shows, that the TestMiniSparkOnYarnCliDriver uses 2 cores, and 2 executor
instances to run the queries. See: https://github.com/apache/hive/blob/master/data/conf/spark/yarn-client/hive-site.xml#L233
> When reading the log files for the query tests, I see the following:
> {code}
> 2017-08-08T07:41:03,315  INFO [0381325d-2c8c-46fb-ab51-423defaddd84 main] session.SparkSession:
Spark cluster current has executors: 1, total cores: 2, memory per executor: 512M, memoryFraction:
0.4
> {code}
> See: http://104.198.109.242/logs/PreCommit-HIVE-Build-6299/succeeded/171-TestMiniSparkOnYarnCliDriver-insert_overwrite_directory2.q-scriptfile1.q-vector_outer_join0.q-and-17-more/logs/hive.log
> When running the tests against a real cluster, I found that running an explain query
for the first time I see 1 executor, but running it for the second time I see 2 executors.
> Also setting some spark configuration on the cluster resets this behavior. For the first
time I will see 1 executor, and for the second time I will see 2 executors again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message