hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20954) Vector RS operator is not using uniform hash function for TPC-DS query 95
Date Wed, 21 Nov 2018 14:18:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16694762#comment-16694762
] 

Hive QA commented on HIVE-20954:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949035/HIVE-20954.1.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 15549 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[keep_uniform] (batchId=78)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets] (batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_smb_mapjoin_14] (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_9]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_subq_exists] (batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_subq_in] (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_subq_not_in] (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cluster] (batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer2]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer3]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer6]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[keep_uniform] (batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3] (batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partialdhj] (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in] (batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in_having] (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi] (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin] (batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar] (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select] (batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_views] (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[table_access_keys_stats]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_reduce_side] (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_union_group_by] (batchId=180)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_auto_smb_mapjoin_14]
(batchId=166)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15027/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15027/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15027/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12949035 - PreCommit-HIVE-Build

> Vector RS operator is not using uniform hash function for TPC-DS query 95
> -------------------------------------------------------------------------
>
>                 Key: HIVE-20954
>                 URL: https://issues.apache.org/jira/browse/HIVE-20954
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Teddy Choi
>            Assignee: Teddy Choi
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: HIVE-20954.1.patch
>
>
> Distribution of rows is skewed in DHJ causing slowdown.
> Same RS outputs, but the two branches use VectorReduceSinkObjectHashOperator and VectorReduceSinkLongOperator.
> {code}
> |                     Select Operator                |
> |                       expressions: ws_warehouse_sk (type: bigint), ws_order_number
(type: bigint) |
> |                       outputColumnNames: _col0, _col1 |
> |                       Select Vectorization:        |
> |                           className: VectorSelectOperator |
> |                           native: true             |
> |                           projectedOutputColumnNums: [14, 16] |
> |                       Statistics: Num rows: 7199963324 Data size: 115185006696 Basic
stats: COMPLETE Column stats: COMPLETE |
> |                       Reduce Output Operator       |
> |                         key expressions: _col1 (type: bigint) |
> |                         sort order: +              |
> |                         Map-reduce partition columns: _col1 (type: bigint) |
> |                         Reduce Sink Vectorization: |
> |                             className: VectorReduceSinkObjectHashOperator |
> |                             keyColumnNums: [16]    |
> |                             native: true           |
> |                             nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled
IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT
columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
|
> |                             partitionColumnNums: [16] |
> |                             valueColumnNums: [14]  |
> +----------------------------------------------------+
> |                      Explain                       |
> +----------------------------------------------------+
> |                         Statistics: Num rows: 7199963324 Data size: 115185006696 Basic
stats: COMPLETE Column stats: COMPLETE |
> |                         value expressions: _col0 (type: bigint) |
> |                       Reduce Output Operator       |
> |                         key expressions: _col1 (type: bigint) |
> |                         sort order: +              |
> |                         Map-reduce partition columns: _col1 (type: bigint) |
> |                         Reduce Sink Vectorization: |
> |                             className: VectorReduceSinkLongOperator |
> |                             keyColumnNums: [16]    |
> |                             native: true           |
> |                             nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled
IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT
columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
|
> |                             valueColumnNums: [14]  |
> |                         Statistics: Num rows: 7199963324 Data size: 115185006696 Basic
stats: COMPLETE Column stats: COMPLETE |
> |                         value expressions: _col0 (type: bigint) |
> |             Execution mode: vectorized, llap       |
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message