hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Vary (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-17292) Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores
Date Thu, 17 Aug 2017 14:53:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Peter Vary updated HIVE-17292:
------------------------------
    Attachment: HIVE-17292.6.patch

The patch contains the following changes:
- Changing Hadoop23Shims.java, so the MiniSparkShim will able to provide the requested 2 executors.
- Changing QTestUtil.setSparkSession, so we will wait until every executor is available, not
only the 1st.
- Changing SparkSessionImpl.getMemoryAndCores, so we use the client provided paralellism in
case of local spark.master too.
- Regenerating golden files (numReducers, and number of files changed in the explain plans)

The change contains 2 golden file changes (spark_dynamic_partition_pruning_mapjoin_only.q.out,
spark_dynamic_partition_pruning.q.out) which are containing other neccessary changes for a
green run, so this patch should be regenerated after their corresponding jiras are solved
(HIVE-17347, HIVE-17346)

> Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores
> ----------------------------------------------------------------------------------
>
>                 Key: HIVE-17292
>                 URL: https://issues.apache.org/jira/browse/HIVE-17292
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark, Test
>    Affects Versions: 3.0.0
>            Reporter: Peter Vary
>            Assignee: Peter Vary
>         Attachments: HIVE-17292.1.patch, HIVE-17292.2.patch, HIVE-17292.3.patch, HIVE-17292.5.patch,
HIVE-17292.6.patch
>
>
> Currently the {{hive-site.xml}} for the {{TestMiniSparkOnYarnCliDriver}} test defines
2 cores, and 2 executors, but only 1 is used, because the MiniCluster does not allows the
creation of the 3rd container.
> The FairScheduler uses 1GB increments for memory, but the containers would like to use
only 512MB. We should change the fairscheduler configuration to use only the requested 512MB



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message