hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14767) Migrate slow MiniMr tests to faster options
Date Thu, 15 Sep 2016 21:18:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15494546#comment-15494546
] 

Prasanth Jayachandran commented on HIVE-14767:
----------------------------------------------

Some tests like skewjoin_onesideskew.q, truncate_column_buckets.q, index_* are slow in MiniLlap
mainly because it spins up an MR job, hence migrated them to run on TestCliDriver.

> Migrate slow MiniMr tests to faster options
> -------------------------------------------
>
>                 Key: HIVE-14767
>                 URL: https://issues.apache.org/jira/browse/HIVE-14767
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Tests
>    Affects Versions: 2.2.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>         Attachments: HIVE-14767.1.patch
>
>
> After analyzing the latest test results https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1200/testReport/org.apache.hadoop.hive.cli/
> there are many Minimr tests that are slow individually. There are also some tests that
does not have to run in minimr, some tests are testing hdfs related features that can be migrated
to MiniLlap (some are already in MiniSpark). Since we have removed hadoop20 now some MiniMr
tests can be moved to TestCliDriver. We should keep the absolute minimum number of tests in
minimr and move other slow test to minillap (spark already has most of these tests).
> These are the individual test runtimes for minimr tests
> {code}
> QFiles	TestMinimrCliDriver elapsed time (Build #1055)
> infer_bucket_sort_reducers_power_two.q	356.211
> bucket5.q	257.589
> infer_bucket_sort_bucketed_table.q	251.557
> bucketizedhiveinputformat.q	249.755
> infer_bucket_sort_map_operators.q	249.484
> index_bitmap_auto.q	212.9
> infer_bucket_sort_dyn_part.q	204.401
> skewjoin_onesideskew.q	198.428
> index_bitmap3.q	194.221
> truncate_column_buckets.q	193.966
> auto_sortmerge_join_16.q	171.094
> schemeAuthority.q	170.839
> leftsemijoin_mr.q	150.493
> join_acid_non_acid.q	143.232
> empty_dir_in_table.q	136.463
> bucketmapjoin6.q	129.786
> reduce_deduplicate.q	116.455
> exchgpartition2lel.q	115.678
> bucket_many.q	104.316
> infer_bucket_sort_merge.q	100.976
> quotedid_smb.q	95.801
> external_table_with_space_in_location_path.q	80.265
> schemeAuthority2.q	72.64
> root_dir_external_table.q	72.171
> parallel_orderby.q	71.662
> disable_merge_for_bucketing.q	69.25
> bucket_num_reducers.q	67.618
> remote_script.q	67.029
> groupby2.q	66.72
> bucket4.q	66.02
> udf_using.q	64.985
> file_with_header_footer.q	42.986
> infer_bucket_sort_num_buckets.q	38.795
> scriptfile1.q	35.317
> bucket6.q	34.963
> non_native_window_udf.q	34.129
> join1.q	33.831
> bucketmapjoin7.q	33.71
> bucket_num_reducers2.q	30.645
> list_bucket_dml_10.q	25.751
> insert_dir_distcp.q	25.723
> uber_reduce.q	16.481
> load_hdfs_file_with_space_in_the_name.q	3.899
> table_nonprintable.q	2.097
> load_fs2.q	1.947
> input16_cc.q	0.916
> temp_table_external.q	0.751
> import_exported_table.q	0.508
> scriptfile1_win.q	0.119
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message