hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chengxiang Li (JIRA)" <>
Subject [jira] [Updated] (HIVE-9211) Research on build mini HoS cluster on YARN for unit test[Spark Branch]
Date Mon, 02 Feb 2015 07:03:35 GMT


Chengxiang Li updated HIVE-9211:
    Attachment: HIVE-9211.7-spark.patch

TestSparkCliDriver launch local spark cluster with \[2,2,1024\], which means 2 executor with
2 cores for each execuotr, while HoS use "spark.executor.cores" values to caculate all cores
number, so TestSparkCliDriver set reduce partition number as 2 instead of 4. Currently caculation
logic of cores number is spark-invaded and easy to be broken, we may handle it in a better
way after SPARK-5080 is resoved. groupby2.q and join1.q is failed due to the previous reason
during EXPLAIN queries, and HIVE-9542 is created for this issue.

ql_rewrite_gbtoidx_cbo_2.q failed on TestMinimrCliDriver as i add result order tag to the
qfile before and did not update TestMinimrCliDriver output.

encryption_join_with_different_encryption_keys.q failure should not related to this patch
from the log file.

> Research on build mini HoS cluster on YARN for unit test[Spark Branch]
> ----------------------------------------------------------------------
>                 Key: HIVE-9211
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Chengxiang Li
>            Assignee: Chengxiang Li
>              Labels: Spark-M5
>         Attachments: HIVE-9211.1-spark.patch, HIVE-9211.2-spark.patch, HIVE-9211.2-spark.patch,
HIVE-9211.3-spark.patch, HIVE-9211.4-spark.patch, HIVE-9211.5-spark.patch, HIVE-9211.6-spark.patch,
> HoS on YARN is a common use case in product environment, we'd better enable unit test
for this case. 

This message was sent by Atlassian JIRA

View raw message