hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chengxiang li" <chengxiang...@intel.com>
Subject Re: Review Request 30264: HIVE-9221 enable unit test for mini Spark on YARN cluster[Spark Branch]
Date Tue, 27 Jan 2015 01:23:48 GMT


> On 一月 26, 2015, 10:30 p.m., Xuefu Zhang wrote:
> > data/conf/spark/yarn-client/hive-site.xml, line 225
> > <https://reviews.apache.org/r/30264/diff/1/?file=834064#file834064line225>
> >
> >     Only one executor? Maybe 2 will make it more general.

Yes, that make sense.


On 一月 26, 2015, 10:30 p.m., chengxiang li wrote:
> > I'm wondering why we have a new set of .out files?

Every Test*CliDriver has its own output directory, i didn't think much about this previously.
With your remind, i think, yes, we could share the golden files with TestSparkCliDriver, as
it's golden file should be the same as TestMiniSparkOnYarnCliDriver for each qtest. One thing
more to be note here is that, as spark.query.files contains more than 500 qtests, and it take
long enough time to run a full unit test for hive now, i didn't enable all spark.query.files
qtest for TestMiniSparkOnYarnCliDriver, instead, i enable qtests from minimr,query.files for
it, which contains about 50 qtests, it takes about 10 minutes in my own desktop.


- chengxiang


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30264/#review69685
-----------------------------------------------------------


On 一月 26, 2015, 6:37 a.m., chengxiang li wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30264/
> -----------------------------------------------------------
> 
> (Updated 一月 26, 2015, 6:37 a.m.)
> 
> 
> Review request for hive, Szehon Ho and Xuefu Zhang.
> 
> 
> Bugs: HIVE-9211
>     https://issues.apache.org/jira/browse/HIVE-9211
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> MiniSparkOnYarnCluster is enabled for unit test, Spark is deployed on miniYarnCluster
on yarn-client mode, all qfiles in minimr.query.files are enabled in this unit test except
3 qfile: bucket_num_reducers.q, bucket_num_reducers2.q, udf_using.q, which is not supported
in HoS.
> 
> 
> Diffs
> -----
> 
>   data/conf/spark/hive-site.xml 016f568 
>   data/conf/spark/standalone/hive-site.xml PRE-CREATION 
>   data/conf/spark/yarn-client/hive-site.xml PRE-CREATION 
>   itests/pom.xml e1e88f6 
>   itests/qtest-spark/pom.xml d12fad5 
>   itests/src/test/resources/testconfiguration.properties f583aaf 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 095b9bd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 41a2ab7

>   ql/src/test/results/clientpositive/miniSparkOnYarn/auto_sortmerge_join_16.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/bucket4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/bucket5.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/bucket6.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/bucketizedhiveinputformat.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin6.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin7.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/constprog_partitioner.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/disable_merge_for_bucketing.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/empty_dir_in_table.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/external_table_with_space_in_location_path.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/file_with_header_footer.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/groupby1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/groupby2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/import_exported_table.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap3.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap_auto.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_bucketed_table.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_dyn_part.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_map_operators.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_merge.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_num_buckets.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_reducers_power_two.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/input16_cc.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/join1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/leftsemijoin_mr.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/list_bucket_dml_10.q.java1.7.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/load_fs2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/load_hdfs_file_with_space_in_the_name.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/parallel_orderby.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_1.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_2.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/quotedid_smb.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/reduce_deduplicate.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/remote_script.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/root_dir_external_table.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority2.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/scriptfile1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/smb_mapjoin_8.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter_partitioned.q.out
PRE-CREATION 
>   ql/src/test/results/clientpositive/miniSparkOnYarn/temp_table_external.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/truncate_column_buckets.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/miniSparkOnYarn/uber_reduce.q.out PRE-CREATION 
>   shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java b17f465

>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java a61c3ac 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java PRE-CREATION

>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 064304c 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java aea90db

> 
> Diff: https://reviews.apache.org/r/30264/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> chengxiang li
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message