hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <>
Subject [jira] [Commented] (HIVE-17270) Qtest results show wrong number of executors
Date Tue, 08 Aug 2017 17:22:00 GMT


Sahil Takiar commented on HIVE-17270:

I'm not sure why the MiniSparkOnYarn cluster shows only 1 executor. My best guess is that
the tests get are started as soon as 1 executor has started (see {{QTestUtil#createSessionState}}).
Its possible the test just finishes before the second executor even gets created.

When running against a real cluster, it depends if {{spark.dynamicAllocation.enabled}} is
set to true or not. If it is true, then the number of executors will be scaled up and down
depending on the resource load.

{{spark.dynamicAllocation.enabled}} is {{false]} by default, so I don't think its enabled
fro the MiniSparkOnYarn tests (although maybe we should change that).

I'm not sure I understand your comment about the reducers. Why should it be 4 instead of 2?

> Qtest results show wrong number of executors
> --------------------------------------------
>                 Key: HIVE-17270
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: 3.0.0
>            Reporter: Peter Vary
>            Assignee: Peter Vary
> The hive-site.xml shows, that the TestMiniSparkOnYarnCliDriver uses 2 cores, and 2 executor
instances to run the queries. See:
> When reading the log files for the query tests, I see the following:
> {code}
> 2017-08-08T07:41:03,315  INFO [0381325d-2c8c-46fb-ab51-423defaddd84 main] session.SparkSession:
Spark cluster current has executors: 1, total cores: 2, memory per executor: 512M, memoryFraction:
> {code}
> See:
> When running the tests against a real cluster, I found that running an explain query
for the first time I see 1 executor, but running it for the second time I see 2 executors.
> Also setting some spark configuration on the cluster resets this behavior. For the first
time I will see 1 executor, and for the second time I will see 2 executors again.

This message was sent by Atlassian JIRA

View raw message