hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang" <xzh...@cloudera.com>
Subject Re: Review Request 30264: HIVE-9221 enable unit test for mini Spark on YARN cluster[Spark Branch]
Date Mon, 26 Jan 2015 22:30:10 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30264/#review69685
-----------------------------------------------------------



data/conf/spark/yarn-client/hive-site.xml
<https://reviews.apache.org/r/30264/#comment114411>

    Only one executor? Maybe 2 will make it more general.


I'm wondering why we have a new set of .out files?

- Xuefu Zhang


On Jan. 26, 2015, 6:37 a.m., chengxiang li wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30264/
> -----------------------------------------------------------
> 
> (Updated Jan. 26, 2015, 6:37 a.m.)
> 
> 
> Review request for hive, Szehon Ho and Xuefu Zhang.
> 
> 
> Bugs: HIVE-9211
>     https://issues.apache.org/jira/browse/HIVE-9211
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> MiniSparkOnYarnCluster is enabled for unit test, Spark is deployed on miniYarnCluster
on yarn-client mode, all qfiles in minimr.query.files are enabled in this unit test except
3 qfile: bucket_num_reducers.q, bucket_num_reducers2.q, udf_using.q, which is not supported
in HoS.
> 
> 
> Diffs
> -----
> 
>   data/conf/spark/hive-site.xml 016f568 
>   data/conf/spark/standalone/hive-site.xml PRE-CREATION 
>   data/conf/spark/yarn-client/hive-site.xml PRE-CREATION 
>   itests/pom.xml e1e88f6 
>   itests/qtest-spark/pom.xml d12fad5 
>   itests/src/test/resources/testconfiguration.properties f583aaf 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 095b9bd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 41a2ab7

>   ql/src/test/results/clientpositive/miniSparkOnYarn/auto_sortmerge_join_16.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/bucket4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/bucket5.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/bucket6.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/bucketizedhiveinputformat.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin6.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin7.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/constprog_partitioner.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/disable_merge_for_bucketing.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/empty_dir_in_table.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/external_table_with_space_in_location_path.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/file_with_header_footer.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/groupby1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/groupby2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/import_exported_table.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap3.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap_auto.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_bucketed_table.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_dyn_part.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_map_operators.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_merge.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_num_buckets.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_reducers_power_two.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/input16_cc.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/join1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/leftsemijoin_mr.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/list_bucket_dml_10.q.java1.7.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/load_fs2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/load_hdfs_file_with_space_in_the_name.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/parallel_orderby.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/quotedid_smb.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/reduce_deduplicate.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/remote_script.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/root_dir_external_table.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority2.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/scriptfile1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/smb_mapjoin_8.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter_partitioned.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/temp_table_external.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/truncate_column_buckets.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/uber_reduce.q.out PRE-CREATION 
>   shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java b17f465

>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java a61c3ac 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java PRE-CREATION

>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 064304c 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java aea90db

> 
> Diff: https://reviews.apache.org/r/30264/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> chengxiang li
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message