hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11634) Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
Date Sat, 31 Oct 2015 03:28:27 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983782#comment-14983782
] 

Hive QA commented on HIVE-11634:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12769833/HIVE-11634.995.patch

{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 9731 tests executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-update_orig_table.q-vectorization_13.q-update_after_multiple_inserts.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.common.metrics.metrics2.TestCodahaleMetrics.testFileReporting
org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5870/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5870/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5870/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12769833 - PreCommit-HIVE-TRUNK-Build

> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> ------------------------------------------------------------------
>
>                 Key: HIVE-11634
>                 URL: https://issues.apache.org/jira/browse/HIVE-11634
>             Project: Hive
>          Issue Type: Bug
>          Components: CBO
>            Reporter: Hari Sankar Sivarama Subramaniyan
>            Assignee: Hari Sankar Sivarama Subramaniyan
>         Attachments: HIVE-11634.1.patch, HIVE-11634.2.patch, HIVE-11634.3.patch, HIVE-11634.4.patch,
HIVE-11634.5.patch, HIVE-11634.6.patch, HIVE-11634.7.patch, HIVE-11634.8.patch, HIVE-11634.9.patch,
HIVE-11634.91.patch, HIVE-11634.92.patch, HIVE-11634.93.patch, HIVE-11634.94.patch, HIVE-11634.95.patch,
HIVE-11634.96.patch, HIVE-11634.97.patch, HIVE-11634.98.patch, HIVE-11634.99.patch, HIVE-11634.990.patch,
HIVE-11634.991.patch, HIVE-11634.992.patch, HIVE-11634.993.patch, HIVE-11634.994.patch, HIVE-11634.995.patch,
HIVE-11634.patch
>
>
> Currently, we do not support partition pruning for the following scenario
> {code}
> create table pcr_t1 (key int, value string) partitioned by (ds string);
> insert overwrite table pcr_t1 partition (ds='2000-04-08') select * from src where key
< 20 order by key;
> insert overwrite table pcr_t1 partition (ds='2000-04-09') select * from src where key
< 20 order by key;
> insert overwrite table pcr_t1 partition (ds='2000-04-10') select * from src where key
< 20 order by key;
> explain extended select ds from pcr_t1 where struct(ds, key) in (struct('2000-04-08',1),
struct('2000-04-09',2));
> {code}
> If we run the above query, we see that all the partitions of table pcr_t1 are present
in the filter predicate where as we can prune  partition (ds='2000-04-10'). 
> The optimization is to rewrite the above query into the following.
> {code}
> explain extended select ds from pcr_t1 where  (struct(ds)) IN (struct('2000-04-08'),
struct('2000-04-09')) and  struct(ds, key) in (struct('2000-04-08',1), struct('2000-04-09',2));
> {code}
> The predicate (struct(ds)) IN (struct('2000-04-08'), struct('2000-04-09'))  is used by
partition pruner to prune the columns which otherwise will not be pruned.
> This is an extension of the idea presented in HIVE-11573.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message