hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
Date Sat, 31 Dec 2016 06:58:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15789063#comment-15789063
] 

Hive QA commented on HIVE-15527:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12845215/HIVE-15527.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 358 failed/errored test(s), 10895 tests executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=233)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] (batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a] (batchId=135)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[auto_sortmerge_join_16]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucket4] (batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucket5] (batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucket6] (batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin6] (batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin7] (batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_partitioner]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_semijoin]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[disable_merge_for_bucketing]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[empty_dir_in_table]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[gen_udf_example_add10]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[index_bitmap3] (batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[index_bitmap_auto] (batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_bucketed_table]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_map_operators]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_merge]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_reducers_power_two]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[leftsemijoin_mr] (batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge_incompat2]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[parallel_orderby] (batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[quotedid_smb] (batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[reduce_deduplicate]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[remote_script] (batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[root_dir_external_table]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[schemeAuthority2] (batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[schemeAuthority] (batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[scriptfile1] (batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[truncate_column_buckets]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join1]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join3]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join4]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join5]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=93)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3] (batchId=92)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4] (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=97)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join0] (batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join15] (batchId=101)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join18] (batchId=99)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join18_multi_distinct] (batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join20] (batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join21] (batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join23] (batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join27] (batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join29] (batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join30] (batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join31] (batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join6] (batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join7] (batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_filters] (batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_nulls] (batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_stats] (batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_without_localtask] (batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_smb_mapjoin_14] (batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_10] (batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_13] (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_14] (batchId=99)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_15] (batchId=99)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_16] (batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_6] (batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_9] (batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[avro_joins] (batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[avro_joins_native] (batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket2] (batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket3] (batchId=101)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket4] (batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_spark4] (batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket_map_join_tez2] (batchId=99)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin13] (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin1] (batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin2] (batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin3] (batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin4] (batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin5] (batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin7] (batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_2] (batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_4] (batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7] (batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_8] (batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_gby] (batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_gby_empty] (batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_limit] (batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_semijoin] (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_simple_select] (batchId=101)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_subq_in] (batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_subq_not_in] (batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_udf_udaf] (batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_union] (batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[column_access_stats] (batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[count] (batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cross_product_check_2] (batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[date_join1] (batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[decimal_1_1] (batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[disable_merge_for_bucketing] (batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[enforce_order] (batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[filter_join_breaktask2] (batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[filter_join_breaktask] (batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby10] (batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby11] (batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby1] (batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby1_map_skew] (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby2] (batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby2_map] (batchId=106)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby2_map_multi_distinct] (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby2_map_skew] (batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby2_noskew] (batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby2_noskew_multi_distinct]
(batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby3] (batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby3_map] (batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby3_map_multi_distinct] (batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby3_map_skew] (batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby3_noskew] (batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby3_noskew_multi_distinct]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby5] (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby6_map_skew] (batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby7] (batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby7_map_multi_single_reducer]
(batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby7_map_skew] (batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby7_noskew_multi_single_reducer]
(batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby8] (batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby8_map] (batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby8_map_skew] (batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby8_noskew] (batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_bigdata] (batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_complex_types_multi_single_reducer]
(batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_cube1] (batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_grouping_id2] (batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_map_ppr] (batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_map_ppr_multi_distinct]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_multi_insert_common_distinct]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_multi_single_reducer2]
(batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_multi_single_reducer3]
(batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_multi_single_reducer]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_position] (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_ppr] (batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_ppr_multi_distinct] (batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_resolution] (batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_rollup1] (batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_sort_1_23] (batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_sort_skew_1_23] (batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[identity_project_remove_skip]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[index_auto_self_join] (batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[innerjoin] (batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[input14] (batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[input17] (batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[input18] (batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[input_part2] (batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[insert_into1] (batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[insert_into2] (batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[insert_into3] (batchId=106)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join0] (batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join10] (batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join11] (batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join12] (batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join13] (batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join14] (batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join15] (batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join17] (batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join18] (batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join18_multi_distinct] (batchId=106)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join20] (batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join21] (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join23] (batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join24] (batchId=106)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join2] (batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join34] (batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join35] (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join3] (batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join41] (batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join4] (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join5] (batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join6] (batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join7] (batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join8] (batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join9] (batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_1to1] (batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_array] (batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_casesensitive] (batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_filters_overlap] (batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_hive_626] (batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_literals] (batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_merge_multi_expressions]
(batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_nullsafe] (batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_rc] (batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_reorder2] (batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_reorder3] (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_reorder] (batchId=99)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_thrift] (batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_vc] (batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[leftsemijoin] (batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[leftsemijoin_mr] (batchId=101)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[limit_pushdown] (batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part2] (batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[louter_join_ppr] (batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin_decimal] (batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin_distinct] (batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin_filter_on_outerjoin] (batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin_mapjoin] (batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin_test_outer] (batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapreduce1] (batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapreduce2] (batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[multi_insert] (batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[multi_insert_gby2] (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[multi_insert_gby3] (batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[multi_insert_gby] (batchId=101)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[multi_insert_lateral_view] (batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[multi_insert_mixed] (batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[multi_insert_move_tasks_share_dependencies]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[multi_insert_with_join] (batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[order] (batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[outer_join_ppr] (batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parallel] (batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parallel_join0] (batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parallel_join1] (batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parquet_join] (batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[pcr] (batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_gby_join] (batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join2] (batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join3] (batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join] (batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join_filter] (batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_multi_insert] (batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_outer_join1] (batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_outer_join2] (batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_outer_join3] (batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_outer_join4] (batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_transform] (batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ptf] (batchId=101)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ptf_decimal] (batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ptf_matchpath] (batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ptf_rcfile] (batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ptf_register_tblfn] (batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ptf_seqfile] (batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ptf_streaming] (batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[router_join_ppr] (batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[runtime_skewjoin_mapjoin_spark]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[sample10] (batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[sample1] (batchId=99)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[sample3] (batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[sample4] (batchId=101)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[sample5] (batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[sample6] (batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[sample7] (batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[sample8] (batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[sample9] (batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[scriptfile1] (batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[semijoin] (batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoin] (batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoin_noskew] (batchId=101)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoin_union_remove_1] (batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoin_union_remove_2] (batchId=106)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt10] (batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt11] (batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt12] (batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt13] (batchId=101)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt14] (batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt15] (batchId=99)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt16] (batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt17] (batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt19] (batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt1] (batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt20] (batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt2] (batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt3] (batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt4] (batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt5] (batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt6] (batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt7] (batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt8] (batchId=106)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[skewjoinopt9] (batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_11] (batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_12] (batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_13] (batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_14] (batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_15] (batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_16] (batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_17] (batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_18] (batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_19] (batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_20] (batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_21] (batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_22] (batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_6] (batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_7] (batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_8] (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[sort] (batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats10] (batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats1] (batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats9] (batchId=106)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_exists] (batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in] (batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multiinsert] (batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[table_access_keys_stats] (batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[temp_table] (batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[temp_table_gb1] (batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[temp_table_join1] (batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[tez_join_tests] (batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[tez_joins_explain] (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[timestamp_lazy] (batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[transform_ppr1] (batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[transform_ppr2] (batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[udaf_collect_set] (batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[udf_percentile] (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union10] (batchId=99)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union12] (batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union17] (batchId=124)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union18] (batchId=101)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union19] (batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union20] (batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union23] (batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union24] (batchId=119)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union26] (batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union27] (batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union28] (batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union29] (batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union30] (batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union31] (batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union32] (batchId=106)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union33] (batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union34] (batchId=99)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union3] (batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union4] (batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union6] (batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_lateralview] (batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_null] (batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_ppr] (batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_23] (batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_6_subq] (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_top_level] (batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[uniquejoin] (batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[varchar_join1] (batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in] (batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_cast_constant] (batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_data_types] (batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_orderby_5] (batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_string_concat] (batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_0] (batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_12] (batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_13] (batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_14] (batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_15] (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_17] (batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_div0] (batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_part] (batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_part_project] (batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress] (batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf] (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_rcfile_columnar] (batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_shufflejoin] (batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_timestamp_funcs] (batchId=107)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[windowing] (batchId=116)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=228)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2752/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2752/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2752/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 358 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12845215 - PreCommit-HIVE-Build

> Memory usage is unbound in SortByShuffler for Spark
> ---------------------------------------------------
>
>                 Key: HIVE-15527
>                 URL: https://issues.apache.org/jira/browse/HIVE-15527
>             Project: Hive
>          Issue Type: Improvement
>          Components: Spark
>    Affects Versions: 1.1.0
>            Reporter: Xuefu Zhang
>            Assignee: Xuefu Zhang
>         Attachments: HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values that have
the same key in shuffled result produced by spark transformation sortByKey. It's possible
that memory can be exhausted because of a large key group.
> {code}
>             @Override
>             public Tuple2<HiveKey, Iterable<BytesWritable>> next() {
>               // TODO: implement this by accumulating rows with the same key into a list.
>               // Note that this list needs to improved to prevent excessive memory usage,
but this
>               // can be done in later phase.
>               while (it.hasNext()) {
>                 Tuple2<HiveKey, BytesWritable> pair = it.next();
>                 if (curKey != null && !curKey.equals(pair._1())) {
>                   HiveKey key = curKey;
>                   List<BytesWritable> values = curValues;
>                   curKey = pair._1();
>                   curValues = new ArrayList<BytesWritable>();
>                   curValues.add(pair._2());
>                   return new Tuple2<HiveKey, Iterable<BytesWritable>>(key,
values);
>                 }
>                 curKey = pair._1();
>                 curValues.add(pair._2());
>               }
>               if (curKey == null) {
>                 throw new NoSuchElementException();
>               }
>               // if we get here, this should be the last element we have
>               HiveKey key = curKey;
>               curKey = null;
>               return new Tuple2<HiveKey, Iterable<BytesWritable>>(key, curValues);
>             }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to backup the
value iterable using the same input iterator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message