Return-Path: X-Original-To: apmail-hive-commits-archive@www.apache.org Delivered-To: apmail-hive-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5A2B511609 for ; Thu, 11 Sep 2014 19:03:16 +0000 (UTC) Received: (qmail 62967 invoked by uid 500); 11 Sep 2014 19:03:16 -0000 Delivered-To: apmail-hive-commits-archive@hive.apache.org Received: (qmail 62919 invoked by uid 500); 11 Sep 2014 19:03:16 -0000 Mailing-List: contact commits-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hive-dev@hive.apache.org Delivered-To: mailing list commits@hive.apache.org Received: (qmail 62907 invoked by uid 99); 11 Sep 2014 19:03:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Sep 2014 19:03:16 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Sep 2014 19:03:14 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 15DC423888D2; Thu, 11 Sep 2014 19:02:54 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1624364 - in /hive/trunk/ql/src: java/org/apache/hadoop/hive/ql/parse/AppMasterEventProcessor.java test/queries/clientpositive/dynamic_partition_pruning_2.q test/results/clientpositive/tez/dynamic_partition_pruning_2.q.out Date: Thu, 11 Sep 2014 19:02:53 -0000 To: commits@hive.apache.org From: gunther@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20140911190254.15DC423888D2@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: gunther Date: Thu Sep 11 19:02:53 2014 New Revision: 1624364 URL: http://svn.apache.org/r1624364 Log: HIVE-8051: Some union queries fail with dynamic partition pruning on tez (Gunther Hagleitner, reviewed by Vikram Dixit K) Modified: hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/AppMasterEventProcessor.java hive/trunk/ql/src/test/queries/clientpositive/dynamic_partition_pruning_2.q hive/trunk/ql/src/test/results/clientpositive/tez/dynamic_partition_pruning_2.q.out Modified: hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/AppMasterEventProcessor.java URL: http://svn.apache.org/viewvc/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/AppMasterEventProcessor.java?rev=1624364&r1=1624363&r2=1624364&view=diff ============================================================================== --- hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/AppMasterEventProcessor.java (original) +++ hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/AppMasterEventProcessor.java Thu Sep 11 19:02:53 2014 @@ -56,6 +56,7 @@ public class AppMasterEventProcessor imp } else { events = new ArrayList(); } + events.add(event); context.tsToEventMap.put(desc.getTableScan(), events); return true; } Modified: hive/trunk/ql/src/test/queries/clientpositive/dynamic_partition_pruning_2.q URL: http://svn.apache.org/viewvc/hive/trunk/ql/src/test/queries/clientpositive/dynamic_partition_pruning_2.q?rev=1624364&r1=1624363&r2=1624364&view=diff ============================================================================== --- hive/trunk/ql/src/test/queries/clientpositive/dynamic_partition_pruning_2.q (original) +++ hive/trunk/ql/src/test/queries/clientpositive/dynamic_partition_pruning_2.q Thu Sep 11 19:02:53 2014 @@ -39,3 +39,12 @@ and d1.label in ('foo', 'bar') GROUP BY d1.label ORDER BY d1.label; + +EXPLAIN +SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'foo' +UNION ALL +SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'bar'; + +SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'foo' +UNION ALL +SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'bar'; \ No newline at end of file Modified: hive/trunk/ql/src/test/results/clientpositive/tez/dynamic_partition_pruning_2.q.out URL: http://svn.apache.org/viewvc/hive/trunk/ql/src/test/results/clientpositive/tez/dynamic_partition_pruning_2.q.out?rev=1624364&r1=1624363&r2=1624364&view=diff ============================================================================== --- hive/trunk/ql/src/test/results/clientpositive/tez/dynamic_partition_pruning_2.q.out (original) +++ hive/trunk/ql/src/test/results/clientpositive/tez/dynamic_partition_pruning_2.q.out Thu Sep 11 19:02:53 2014 @@ -267,3 +267,176 @@ POSTHOOK: Input: default@dim_shops #### A masked pattern was here #### bar 3 15 foo 3 6 +PREHOOK: query: EXPLAIN +SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'foo' +UNION ALL +SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'bar' +PREHOOK: type: QUERY +POSTHOOK: query: EXPLAIN +SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'foo' +UNION ALL +SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'bar' +POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-1 is a root stage + Stage-0 depends on stages: Stage-1 + +STAGE PLANS: + Stage: Stage-1 + Tez + Edges: + Map 3 <- Map 1 (BROADCAST_EDGE), Union 4 (CONTAINS) + Map 5 <- Map 2 (BROADCAST_EDGE), Union 4 (CONTAINS) +#### A masked pattern was here #### + Vertices: + Map 1 + Map Operator Tree: + TableScan + alias: dim_shops + filterExpr: (id is not null and (label = 'bar')) (type: boolean) + Statistics: Num rows: 2 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: (id is not null and (label = 'bar')) (type: boolean) + Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE + Reduce Output Operator + key expressions: id (type: int) + sort order: + + Map-reduce partition columns: id (type: int) + Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE + Select Operator + expressions: id (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE + Dynamic Partitioning Event Operator + Target Input: agg_01 + Partition key expr: dim_shops_id + Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE + Target column: dim_shops_id + Target Vertex: Map 3 + Map 2 + Map Operator Tree: + TableScan + alias: dim_shops + filterExpr: (id is not null and (label = 'foo')) (type: boolean) + Statistics: Num rows: 2 Data size: 18 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: (id is not null and (label = 'foo')) (type: boolean) + Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE + Reduce Output Operator + key expressions: id (type: int) + sort order: + + Map-reduce partition columns: id (type: int) + Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE + Select Operator + expressions: id (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE + Group By Operator + keys: _col0 (type: int) + mode: hash + outputColumnNames: _col0 + Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE + Dynamic Partitioning Event Operator + Target Input: agg_01 + Partition key expr: dim_shops_id + Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE + Target column: dim_shops_id + Target Vertex: Map 5 + Map 3 + Map Operator Tree: + TableScan + alias: agg_01 + filterExpr: dim_shops_id is not null (type: boolean) + Map Join Operator + condition map: + Inner Join 0 to 1 + condition expressions: + 0 {amount} {dim_shops_id} + 1 {id} + keys: + 0 dim_shops_id (type: int) + 1 id (type: int) + outputColumnNames: _col0, _col1, _col5 + Filter Operator + predicate: (_col1 = _col5) (type: boolean) + Select Operator + expressions: _col0 (type: decimal(10,0)) + outputColumnNames: _col0 + Select Operator + expressions: _col0 (type: decimal(10,0)) + outputColumnNames: _col0 + File Output Operator + compressed: false + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + Map 5 + Map Operator Tree: + TableScan + alias: agg_01 + filterExpr: dim_shops_id is not null (type: boolean) + Map Join Operator + condition map: + Inner Join 0 to 1 + condition expressions: + 0 {amount} {dim_shops_id} + 1 {id} + keys: + 0 dim_shops_id (type: int) + 1 id (type: int) + outputColumnNames: _col0, _col1, _col5 + Filter Operator + predicate: (_col1 = _col5) (type: boolean) + Select Operator + expressions: _col0 (type: decimal(10,0)) + outputColumnNames: _col0 + Select Operator + expressions: _col0 (type: decimal(10,0)) + outputColumnNames: _col0 + File Output Operator + compressed: false + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + Union 4 + Vertex: Union 4 + + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + ListSink + +PREHOOK: query: SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'foo' +UNION ALL +SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'bar' +PREHOOK: type: QUERY +PREHOOK: Input: default@agg_01 +PREHOOK: Input: default@agg_01@dim_shops_id=1 +PREHOOK: Input: default@agg_01@dim_shops_id=2 +PREHOOK: Input: default@agg_01@dim_shops_id=3 +PREHOOK: Input: default@dim_shops +#### A masked pattern was here #### +POSTHOOK: query: SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'foo' +UNION ALL +SELECT amount FROM agg_01, dim_shops WHERE dim_shops_id = id AND label = 'bar' +POSTHOOK: type: QUERY +POSTHOOK: Input: default@agg_01 +POSTHOOK: Input: default@agg_01@dim_shops_id=1 +POSTHOOK: Input: default@agg_01@dim_shops_id=2 +POSTHOOK: Input: default@agg_01@dim_shops_id=3 +POSTHOOK: Input: default@dim_shops +#### A masked pattern was here #### +4 +5 +6 +1 +2 +3