hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-15121) Last MR job in Hive should be able to write to a different scratch directory
Date Fri, 04 Nov 2016 22:04:59 GMT

    [ https://issues.apache.org/jira/browse/HIVE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15637848#comment-15637848
] 

Hive QA commented on HIVE-15121:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12837281/HIVE-15121.WIP.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 153 failed/errored test(s), 10628 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=215)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_table] (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5] (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_6] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[binary_output_format] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin5] (batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative2] (batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative] (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[case_sensitivity] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cast1] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_dp] (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_type] (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_colname] (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_uses_database_location] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_3] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_3] (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_4] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_5] (batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_rdd_cache] (batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_ddl] (batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_1_23] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_3] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_5] (batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_7] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_skew_1_23] (batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_test_1] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_auto] (batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_auto_file_format] (batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_auto_mult_tables_compact] (batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_auto_multiple] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_auto_partitioned] (batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_auto_update] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_compression] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_serde] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_skewtable] (batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input11] (batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input12] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input13] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input34] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input35] (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input36] (batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input38] (batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input6] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input7] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input8] (batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input9] (batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_dynamicserde] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_part1] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_part2] (batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_part5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_testsequencefile] (batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_testxpath2] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_testxpath] (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert1] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_into1] (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_into4] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_into5] (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_into6] (batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join25] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join26] (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join27] (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join36] (batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join37] (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join39] (batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_map_ppr] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lineage1] (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part14] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part1] (batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[merge1] (batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nullformatCTAS] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_createas1] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge2] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge3] (batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcr] (batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_constant_expr] (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[quote1] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[rand_partitionpruner2] (batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample1] (batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample5] (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin9] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_7] (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[special_character_in_tabnames_2] (batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats4] (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_empty_dyn_part] (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subq] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_alias] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf1] (batchId=76)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_10_trims] (batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_length] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_reverse] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union10] (batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union12] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union18] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union28] (batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union29] (batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union30] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union33] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union4] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union6] (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_stats] (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_4] (batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_multi_insert] (batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join1] (batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join2] (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join3] (batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join4] (batchId=76)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_tablesample_rows] (batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_varchar_4] (batchId=27)
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[serde_typedbytes2] (batchId=218)
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[serde_typedbytes3] (batchId=218)
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[serde_typedbytes5] (batchId=218)
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[serde_typedbytes] (batchId=218)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_bulk] (batchId=89)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_single_sourced_multi_insert]
(batchId=88)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[bucket5] (batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge10] (batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge1] (batchId=131)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge_diff_fs] (batchId=131)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid] (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge5] (batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge6] (batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge7] (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_merge9] (batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats] (batchId=145)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge1] (batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge2] (batchId=158)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge3] (batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge4] (batchId=157)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge5] (batchId=157)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge6] (batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge7] (batchId=158)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge9] (batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge_diff_fs] (batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3] (batchId=90)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=90)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_map_operators]
(batchId=81)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[merge1] (batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[merge2] (batchId=92)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_10] (batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_11] (batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_12] (batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_13] (batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_14] (batchId=97)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_16] (batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_4] (batchId=96)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_5] (batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_9] (batchId=123)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1972/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1972/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1972/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 153 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12837281 - PreCommit-HIVE-Build

> Last MR job in Hive should be able to write to a different scratch directory
> ----------------------------------------------------------------------------
>
>                 Key: HIVE-15121
>                 URL: https://issues.apache.org/jira/browse/HIVE-15121
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Hive
>            Reporter: Sahil Takiar
>         Attachments: HIVE-15121.WIP.1.patch, HIVE-15121.WIP.2.patch, HIVE-15121.WIP.patch
>
>
> Hive should be able to configure all intermediate MR jobs to write to HDFS, but the final
MR job to write to S3.
> This will be useful for implementing parallel renames on S3. The idea is that for a mutli-job
query, all intermediate MR jobs write to HDFS, and then the final job writes to S3. Writing
to HDFS should be faster than writing to S3, so it makes more sense to write intermediate
data to HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message