hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14016) Vectorization: VectorGroupByRollupOperator and VectorGroupByCubeOperator
Date Wed, 25 Jan 2017 15:17:26 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837887#comment-15837887
] 

Hive QA commented on HIVE-14016:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12849274/HIVE-14016.01.patch

{color:green}SUCCESS:{color} +1 due to 19 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 11011 tests executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_char_mapjoin1] (batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_count] (batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_empty_where] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_grouping_sets] (batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_varchar_mapjoin1] (batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_15] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_limit] (batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_distinct_gby] (batchId=68)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[offset_limit_ppd_optimizer]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets2]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets4]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets5]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_15] (batchId=151)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=93)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_15] (batchId=122)
org.apache.hive.hcatalog.pig.TestRCFileHCatStorer.testWriteDecimalX (batchId=173)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteDate (batchId=173)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteDecimalX (batchId=173)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3174/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3174/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3174/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12849274 - PreCommit-HIVE-Build

> Vectorization: VectorGroupByRollupOperator and VectorGroupByCubeOperator
> ------------------------------------------------------------------------
>
>                 Key: HIVE-14016
>                 URL: https://issues.apache.org/jira/browse/HIVE-14016
>             Project: Hive
>          Issue Type: Improvement
>          Components: Vectorization
>            Reporter: Gopal V
>            Assignee: Matt McCline
>         Attachments: HIVE-14016.01.patch
>
>
> Rollup and Cube queries are not vectorized today due to the miss of grouping-sets inside
vector group by.
> The cube and rollup operators can be shimmed onto the end of the pipeline by converting
a single row writer into a multiple row writer.
> The corresponding non-vec loop is as follows
> {code}
>       if (groupingSetsPresent) {
>         Object[] newKeysArray = newKeys.getKeyArray();
>         Object[] cloneNewKeysArray = new Object[newKeysArray.length];
>         for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
>           cloneNewKeysArray[keyPos] = newKeysArray[keyPos];
>         }
>         for (int groupingSetPos = 0; groupingSetPos < groupingSets.size(); groupingSetPos++)
{
>           for (int keyPos = 0; keyPos < groupingSetsPosition; keyPos++) {
>             newKeysArray[keyPos] = null;
>           }
>           FastBitSet bitset = groupingSetsBitSet[groupingSetPos];
>           // Some keys need to be left to null corresponding to that grouping set.
>           for (int keyPos = bitset.nextSetBit(0); keyPos >= 0;
>             keyPos = bitset.nextSetBit(keyPos+1)) {
>             newKeysArray[keyPos] = cloneNewKeysArray[keyPos];
>           }
>           newKeysArray[groupingSetsPosition] = newKeysGroupingSets[groupingSetPos];
>           processKey(row, rowInspector);
>         }
>       }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message