hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-22785) Update/delete/merge statements not optimized through CBO
Date Thu, 05 Mar 2020 12:57:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052106#comment-17052106
] 

Hive QA commented on HIVE-22785:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995661/HIVE-22785.3.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 134 failed/errored test(s), 18101 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[map_join_on_filter] (batchId=309)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_view_delete] (batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_disable_cbo_1] (batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join0] (batchId=102)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join15] (batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join20] (batchId=103)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join21] (batchId=94)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join23] (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join28] (batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join29] (batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join31] (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join0] (batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer14] (batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_char] (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_date] (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_varchar] (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_non_partitioned] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_orig_table] (batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_whole_partition] (batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid2] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[escape_sortby1] (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[identity_project_remove_skip] (batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input4_limit] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_part7] (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_update_delete] (batchId=99)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join0] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join15] (batchId=94)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join20] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join21] (batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join23] (batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join40] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_filter_on_outerjoin] (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_test_outer] (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[no_hooks] (batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcs] (batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join4] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[print_header] (batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_ppr] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_non_partitioned] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_two_cols] (batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_delete_orig_table] (batchId=2)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets] (batchId=187)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join0] (batchId=193)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join21] (batchId=191)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join29] (batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_join30] (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_num_reducers_acid2]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_num_reducers_acid]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint] (batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ctas] (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction_3]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1] (batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[identity_project_remove_skip]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_into_default_keyword]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join0] (batchId=184)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=191)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_3]
(batchId=180)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_predicate_pushdown]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[parquet_predicate_pushdown]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[runtime_stats_merge] (batchId=191)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part_update_llap_io]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_table_update]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acidvec_part_update_llap_io]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acidvec_table_update_llap_io]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin] (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sqlmerge] (batchId=191)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sqlmerge_stats] (batchId=190)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[unionDistinct_3] (batchId=180)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[update_all_types] (batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[update_where_non_partitioned]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[update_where_partitioned]
(batchId=185)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_acid4] (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_join30] (batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_leftsemi_mapjoin]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_mapjoin_complex_values]
(batchId=185)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_partitioned_date_time]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[dynamic_rdd_cache] (batchId=201)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=201)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=117)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] (batchId=116)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_cbo_rewrite]
(batchId=106)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_no_cbo_rewrite_2]
(batchId=107)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[update_notnull_constraint]
(batchId=106)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join0] (batchId=158)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join15] (batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join20] (batchId=159)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join21] (batchId=156)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join23] (batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join28] (batchId=151)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join29] (batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join30] (batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join31] (batchId=139)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ctas] (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[identity_project_remove_skip]
(batchId=141)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join0] (batchId=145)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join15] (batchId=156)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join20] (batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join21] (batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join23] (batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join40] (batchId=142)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin_filter_on_outerjoin] (batchId=146)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin_test_outer] (batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[parallel_join0] (batchId=152)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join4] (batchId=144)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[semijoin] (batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[sort] (batchId=150)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_ppr] (batchId=126)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCrudMajorCompactionSplitGrouper (batchId=253)
org.apache.hadoop.hive.ql.TestTxnCommands.testDeleteIn (batchId=362)
org.apache.hadoop.hive.ql.TestTxnCommands.testMoreBucketsThanReducers (batchId=362)
org.apache.hadoop.hive.ql.TestTxnCommands.testMultipleDelete (batchId=362)
org.apache.hadoop.hive.ql.TestTxnCommands.testUpdateDeleteOfInserts (batchId=362)
org.apache.hadoop.hive.ql.TestTxnCommands.testUpdateOfInserts (batchId=362)
org.apache.hadoop.hive.ql.TestTxnCommands2.testACIDwithSchemaEvolutionAndCompaction (batchId=343)
org.apache.hadoop.hive.ql.TestTxnCommands2.testDeleteIn (batchId=343)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testACIDwithSchemaEvolutionAndCompaction
(batchId=357)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testDeleteIn (batchId=357)
org.apache.hadoop.hive.ql.TestTxnCommands3.testDeleteEventPruningOff (batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands3.testDeleteEventPruningOn (batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testDeleteIn (batchId=343)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMultipleDelete
(batchId=343)
org.apache.hadoop.hive.ql.TestTxnConcatenate.testConcatenate (batchId=327)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdate (batchId=318)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdateVectorized (batchId=318)
org.apache.hive.jdbc.TestJdbcDriver2.testExceucteUpdateCounts (batchId=292)
org.apache.hive.minikdc.TestSSLWithMiniKdc.testConnection (batchId=307)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20962/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20962/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20962/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 134 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12995661 - PreCommit-HIVE-Build

> Update/delete/merge statements not optimized through CBO
> --------------------------------------------------------
>
>                 Key: HIVE-22785
>                 URL: https://issues.apache.org/jira/browse/HIVE-22785
>             Project: Hive
>          Issue Type: Improvement
>          Components: CBO
>            Reporter: Jesus Camacho Rodriguez
>            Assignee: Krisztian Kasa
>            Priority: Critical
>         Attachments: HIVE-22785.1.patch, HIVE-22785.2.patch, HIVE-22785.2.patch, HIVE-22785.3.patch
>
>
> Currently, CBO is bypassed for update/delete/merge statements.
> To support optimizing these statements through CBO, we need to complete three main tasks:
1) support for sort in Calcite planner, 2) support for SORT in AST converter, and 3) {{RewriteSemanticAnalyzer}}
should extend {{CalcitePlanner}} instead of {{SemanticAnalyzer}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message