From commits-return-31108-archive-asf-public=cust-asf.ponee.io@hive.apache.org Fri Jan 5 11:48:58 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 03F1E180780 for ; Fri, 5 Jan 2018 11:48:58 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id E89D4160C3B; Fri, 5 Jan 2018 10:48:57 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id DCE9F160C19 for ; Fri, 5 Jan 2018 11:48:55 +0100 (CET) Received: (qmail 63517 invoked by uid 500); 5 Jan 2018 10:48:53 -0000 Mailing-List: contact commits-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hive-dev@hive.apache.org Delivered-To: mailing list commits@hive.apache.org Received: (qmail 63212 invoked by uid 99); 5 Jan 2018 10:48:52 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Jan 2018 10:48:52 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id E996FF1826; Fri, 5 Jan 2018 10:48:51 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: kgyrtkirk@apache.org To: commits@hive.apache.org Date: Fri, 05 Jan 2018 10:49:04 -0000 Message-Id: <30bdd740fddd46e5acad97ba920e3f7d@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [15/18] hive git commit: HIVE-18108: in case basic stats are missing; rowcount estimation depends on the selected columns size (Zoltan Haindrich, reviewed by Ashutosh Chauhan) http://git-wip-us.apache.org/repos/asf/hive/blob/d0fa7d55/ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out ---------------------------------------------------------------------- diff --git a/ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out b/ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out index 010f05d..8801396 100644 --- a/ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out +++ b/ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out @@ -136,7 +136,7 @@ POSTHOOK: type: LOAD POSTHOOK: Output: default@bucket_medium@ds=2008-04-08 Warning: Map Join MAPJOIN[32][bigTable=?] in task 'Stage-8:MAPRED' is a cross product Warning: Map Join MAPJOIN[31][bigTable=?] in task 'Stage-7:MAPRED' is a cross product -Warning: Shuffle Join JOIN[17][tables = [$hdt$_1, $hdt$_2, $hdt$_0, $hdt$_3]] in Stage 'Stage-2:MAPRED' is a cross product +Warning: Shuffle Join JOIN[17][tables = [$hdt$_0, $hdt$_1, $hdt$_2, $hdt$_3]] in Stage 'Stage-2:MAPRED' is a cross product PREHOOK: query: explain extended select count(*) FROM bucket_small a JOIN bucket_medium b ON a.key = b.key JOIN bucket_big c ON c.key = b.key JOIN bucket_medium d ON c.key = b.key PREHOOK: type: QUERY POSTHOOK: query: explain extended select count(*) FROM bucket_small a JOIN bucket_medium b ON a.key = b.key JOIN bucket_big c ON c.key = b.key JOIN bucket_medium d ON c.key = b.key @@ -166,7 +166,7 @@ STAGE PLANS: Stage: Stage-16 Map Reduce Local Work Alias -> Map Local Tables: - $hdt$_0:c + $hdt$_1:b Fetch Operator limit: -1 Partition Description: @@ -177,71 +177,23 @@ STAGE PLANS: partition values: ds 2008-04-08 properties: - bucket_count 4 - bucket_field_name key - column.name.delimiter , - columns key,value - columns.comments - columns.types string:string -#### A masked pattern was here #### - name default.bucket_big - numFiles 4 - numRows 0 - partition_columns ds - partition_columns.types string - rawDataSize 0 - serialization.ddl struct bucket_big { string key, string value} - serialization.format 1 - serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - totalSize 5812 -#### A masked pattern was here #### - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - properties: - SORTBUCKETCOLSPREFIX TRUE - bucket_count 4 - bucket_field_name key - column.name.delimiter , - columns key,value - columns.comments - columns.types string:string -#### A masked pattern was here #### - name default.bucket_big - partition_columns ds - partition_columns.types string - serialization.ddl struct bucket_big { string key, string value} - serialization.format 1 - serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe -#### A masked pattern was here #### - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - name: default.bucket_big - name: default.bucket_big - Partition - base file name: ds=2008-04-09 - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - partition values: - ds 2008-04-09 - properties: - bucket_count 4 + bucket_count 3 bucket_field_name key column.name.delimiter , columns key,value columns.comments columns.types string:string #### A masked pattern was here #### - name default.bucket_big - numFiles 4 + name default.bucket_medium + numFiles 3 numRows 0 partition_columns ds partition_columns.types string rawDataSize 0 - serialization.ddl struct bucket_big { string key, string value} + serialization.ddl struct bucket_medium { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - totalSize 5812 + totalSize 170 #### A masked pattern was here #### serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -249,24 +201,24 @@ STAGE PLANS: output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: SORTBUCKETCOLSPREFIX TRUE - bucket_count 4 + bucket_count 3 bucket_field_name key column.name.delimiter , columns key,value columns.comments columns.types string:string #### A masked pattern was here #### - name default.bucket_big + name default.bucket_medium partition_columns ds partition_columns.types string - serialization.ddl struct bucket_big { string key, string value} + serialization.ddl struct bucket_medium { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe #### A masked pattern was here #### serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - name: default.bucket_big - name: default.bucket_big - $hdt$_2:b + name: default.bucket_medium + name: default.bucket_medium + $hdt$_2:a Fetch Operator limit: -1 Partition Description: @@ -277,23 +229,23 @@ STAGE PLANS: partition values: ds 2008-04-08 properties: - bucket_count 3 + bucket_count 2 bucket_field_name key column.name.delimiter , columns key,value columns.comments columns.types string:string #### A masked pattern was here #### - name default.bucket_medium - numFiles 3 + name default.bucket_small + numFiles 2 numRows 0 partition_columns ds partition_columns.types string rawDataSize 0 - serialization.ddl struct bucket_medium { string key, string value} + serialization.ddl struct bucket_small { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - totalSize 170 + totalSize 114 #### A masked pattern was here #### serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -301,56 +253,56 @@ STAGE PLANS: output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: SORTBUCKETCOLSPREFIX TRUE - bucket_count 3 + bucket_count 2 bucket_field_name key column.name.delimiter , columns key,value columns.comments columns.types string:string #### A masked pattern was here #### - name default.bucket_medium + name default.bucket_small partition_columns ds partition_columns.types string - serialization.ddl struct bucket_medium { string key, string value} + serialization.ddl struct bucket_small { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe #### A masked pattern was here #### serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - name: default.bucket_medium - name: default.bucket_medium + name: default.bucket_small + name: default.bucket_small Alias -> Map Local Operator Tree: - $hdt$_0:c + $hdt$_1:b TableScan - alias: c - Statistics: Num rows: 116 Data size: 11624 Basic stats: COMPLETE Column stats: NONE + alias: b + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 116 Data size: 11624 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 116 Data size: 11624 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) 2 _col0 (type: string) Position of Big Table: 0 - $hdt$_2:b + $hdt$_2:a TableScan - alias: b - Statistics: Num rows: 1 Data size: 170 Basic stats: COMPLETE Column stats: NONE + alias: a + Statistics: Num rows: 2 Data size: 1140 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 1 Data size: 170 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2 Data size: 1140 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 170 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2 Data size: 1140 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) @@ -362,17 +314,17 @@ STAGE PLANS: Map Reduce Map Operator Tree: TableScan - alias: a - Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE + alias: c + Statistics: Num rows: 240 Data size: 116240 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 240 Data size: 116240 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 240 Data size: 116240 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -382,7 +334,7 @@ STAGE PLANS: 1 _col0 (type: string) 2 _col0 (type: string) Position of Big Table: 0 - Statistics: Num rows: 255 Data size: 25572 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 528 Data size: 255728 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false GlobalTableId: 0 @@ -603,7 +555,8 @@ STAGE PLANS: name: default.bucket_small name: default.bucket_small Truncated Path -> Alias: - /bucket_small/ds=2008-04-08 [$hdt$_1:a] + /bucket_big/ds=2008-04-08 [$hdt$_0:c] + /bucket_big/ds=2008-04-09 [$hdt$_0:c] Stage: Stage-9 Conditional Operator @@ -667,10 +620,10 @@ STAGE PLANS: $hdt$_3:d TableScan alias: d - Statistics: Num rows: 1 Data size: 1700 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE GatherStats: false Select Operator - Statistics: Num rows: 1 Data size: 1700 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 @@ -689,12 +642,12 @@ STAGE PLANS: 0 1 Position of Big Table: 0 - Statistics: Num rows: 255 Data size: 459327 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 1584 Data size: 1666368 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() mode: hash outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 8 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false GlobalTableId: 0 @@ -800,7 +753,7 @@ STAGE PLANS: Reduce Output Operator null sort order: sort order: - Statistics: Num rows: 1 Data size: 8 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE tag: -1 value expressions: _col0 (type: bigint) auto parallelism: false @@ -837,13 +790,13 @@ STAGE PLANS: aggregations: count(VALUE._col0) mode: mergepartial outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 8 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false GlobalTableId: 0 #### A masked pattern was here #### NumFilesPerFileSink: 1 - Statistics: Num rows: 1 Data size: 8 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE #### A masked pattern was here #### table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat @@ -882,10 +835,10 @@ STAGE PLANS: Map Operator Tree: TableScan alias: d - Statistics: Num rows: 1 Data size: 1700 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE GatherStats: false Select Operator - Statistics: Num rows: 1 Data size: 1700 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -893,12 +846,12 @@ STAGE PLANS: 0 1 Position of Big Table: 1 - Statistics: Num rows: 255 Data size: 459327 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 1584 Data size: 1666368 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() mode: hash outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 8 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false GlobalTableId: 0 @@ -1004,19 +957,19 @@ STAGE PLANS: Reduce Output Operator null sort order: sort order: - Statistics: Num rows: 255 Data size: 25572 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 528 Data size: 255728 Basic stats: COMPLETE Column stats: NONE tag: 0 auto parallelism: false TableScan alias: d - Statistics: Num rows: 1 Data size: 1700 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE GatherStats: false Select Operator - Statistics: Num rows: 1 Data size: 1700 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator null sort order: sort order: - Statistics: Num rows: 1 Data size: 1700 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE tag: 1 auto parallelism: false Path -> Alias: @@ -1104,12 +1057,12 @@ STAGE PLANS: keys: 0 1 - Statistics: Num rows: 255 Data size: 459327 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 1584 Data size: 1666368 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() mode: hash outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 8 Basic stats: PARTIAL Column stats: NONE + Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false GlobalTableId: 0 @@ -1232,7 +1185,7 @@ STAGE PLANS: serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.bucket_big name: default.bucket_big - $hdt$_1:a + $hdt$_2:a Fetch Operator limit: -1 Partition Description: @@ -1288,35 +1241,35 @@ STAGE PLANS: $hdt$_0:c TableScan alias: c - Statistics: Num rows: 116 Data size: 11624 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 240 Data size: 116240 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 116 Data size: 11624 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 240 Data size: 116240 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 116 Data size: 11624 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 240 Data size: 116240 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) 2 _col0 (type: string) Position of Big Table: 1 - $hdt$_1:a + $hdt$_2:a TableScan alias: a - Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2 Data size: 1140 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2 Data size: 1140 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2 Data size: 1140 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) @@ -1329,16 +1282,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: b - Statistics: Num rows: 1 Data size: 170 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 1 Data size: 170 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 170 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -1348,7 +1301,7 @@ STAGE PLANS: 1 _col0 (type: string) 2 _col0 (type: string) Position of Big Table: 1 - Statistics: Num rows: 255 Data size: 25572 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 528 Data size: 255728 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false GlobalTableId: 0 @@ -1569,12 +1522,12 @@ STAGE PLANS: name: default.bucket_small name: default.bucket_small Truncated Path -> Alias: - /bucket_medium/ds=2008-04-08 [$hdt$_2:b] + /bucket_medium/ds=2008-04-08 [$hdt$_1:b] Stage: Stage-18 Map Reduce Local Work Alias -> Map Local Tables: - $hdt$_1:a + $hdt$_0:c Fetch Operator limit: -1 Partition Description: @@ -1585,23 +1538,23 @@ STAGE PLANS: partition values: ds 2008-04-08 properties: - bucket_count 2 + bucket_count 4 bucket_field_name key column.name.delimiter , columns key,value columns.comments columns.types string:string #### A masked pattern was here #### - name default.bucket_small - numFiles 2 + name default.bucket_big + numFiles 4 numRows 0 partition_columns ds partition_columns.types string rawDataSize 0 - serialization.ddl struct bucket_small { string key, string value} + serialization.ddl struct bucket_big { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - totalSize 114 + totalSize 5812 #### A masked pattern was here #### serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1609,24 +1562,72 @@ STAGE PLANS: output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat properties: SORTBUCKETCOLSPREFIX TRUE - bucket_count 2 + bucket_count 4 bucket_field_name key column.name.delimiter , columns key,value columns.comments columns.types string:string #### A masked pattern was here #### - name default.bucket_small + name default.bucket_big partition_columns ds partition_columns.types string - serialization.ddl struct bucket_small { string key, string value} + serialization.ddl struct bucket_big { string key, string value} serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe #### A masked pattern was here #### serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - name: default.bucket_small - name: default.bucket_small - $hdt$_2:b + name: default.bucket_big + name: default.bucket_big + Partition + base file name: ds=2008-04-09 + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + partition values: + ds 2008-04-09 + properties: + bucket_count 4 + bucket_field_name key + column.name.delimiter , + columns key,value + columns.comments + columns.types string:string +#### A masked pattern was here #### + name default.bucket_big + numFiles 4 + numRows 0 + partition_columns ds + partition_columns.types string + rawDataSize 0 + serialization.ddl struct bucket_big { string key, string value} + serialization.format 1 + serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + totalSize 5812 +#### A masked pattern was here #### + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + properties: + SORTBUCKETCOLSPREFIX TRUE + bucket_count 4 + bucket_field_name key + column.name.delimiter , + columns key,value + columns.comments + columns.types string:string +#### A masked pattern was here #### + name default.bucket_big + partition_columns ds + partition_columns.types string + serialization.ddl struct bucket_big { string key, string value} + serialization.format 1 + serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe +#### A masked pattern was here #### + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + name: default.bucket_big + name: default.bucket_big + $hdt$_1:b Fetch Operator limit: -1 Partition Description: @@ -1679,38 +1680,38 @@ STAGE PLANS: name: default.bucket_medium name: default.bucket_medium Alias -> Map Local Operator Tree: - $hdt$_1:a + $hdt$_0:c TableScan - alias: a - Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE + alias: c + Statistics: Num rows: 240 Data size: 116240 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 240 Data size: 116240 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 240 Data size: 116240 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) 1 _col0 (type: string) 2 _col0 (type: string) Position of Big Table: 2 - $hdt$_2:b + $hdt$_1:b TableScan alias: b - Statistics: Num rows: 1 Data size: 170 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 1 Data size: 170 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 170 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) @@ -1722,17 +1723,17 @@ STAGE PLANS: Map Reduce Map Operator Tree: TableScan - alias: c - Statistics: Num rows: 116 Data size: 11624 Basic stats: COMPLETE Column stats: NONE + alias: a + Statistics: Num rows: 2 Data size: 1140 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 116 Data size: 11624 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2 Data size: 1140 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 116 Data size: 11624 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2 Data size: 1140 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -1742,7 +1743,7 @@ STAGE PLANS: 1 _col0 (type: string) 2 _col0 (type: string) Position of Big Table: 2 - Statistics: Num rows: 255 Data size: 25572 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 528 Data size: 255728 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false GlobalTableId: 0 @@ -1963,70 +1964,69 @@ STAGE PLANS: name: default.bucket_small name: default.bucket_small Truncated Path -> Alias: - /bucket_big/ds=2008-04-08 [$hdt$_0:c] - /bucket_big/ds=2008-04-09 [$hdt$_0:c] + /bucket_small/ds=2008-04-08 [$hdt$_2:a] Stage: Stage-1 Map Reduce Map Operator Tree: TableScan - alias: a - Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE + alias: c + Statistics: Num rows: 240 Data size: 116240 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 240 Data size: 116240 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 240 Data size: 116240 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) null sort order: a sort order: + Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 240 Data size: 116240 Basic stats: COMPLETE Column stats: NONE tag: 0 auto parallelism: false TableScan alias: b - Statistics: Num rows: 1 Data size: 170 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 1 Data size: 170 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 170 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) null sort order: a sort order: + Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 1 Data size: 170 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 1700 Basic stats: COMPLETE Column stats: NONE tag: 1 auto parallelism: false TableScan - alias: c - Statistics: Num rows: 116 Data size: 11624 Basic stats: COMPLETE Column stats: NONE + alias: a + Statistics: Num rows: 2 Data size: 1140 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 116 Data size: 11624 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2 Data size: 1140 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 116 Data size: 11624 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2 Data size: 1140 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string) null sort order: a sort order: + Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 116 Data size: 11624 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2 Data size: 1140 Basic stats: COMPLETE Column stats: NONE tag: 2 auto parallelism: false Path -> Alias: @@ -2231,8 +2231,8 @@ STAGE PLANS: Truncated Path -> Alias: /bucket_big/ds=2008-04-08 [$hdt$_0:c] /bucket_big/ds=2008-04-09 [$hdt$_0:c] - /bucket_medium/ds=2008-04-08 [$hdt$_2:b] - /bucket_small/ds=2008-04-08 [$hdt$_1:a] + /bucket_medium/ds=2008-04-08 [$hdt$_1:b] + /bucket_small/ds=2008-04-08 [$hdt$_2:a] Needs Tagging: true Reduce Operator Tree: Join Operator @@ -2243,7 +2243,7 @@ STAGE PLANS: 0 _col0 (type: string) 1 _col0 (type: string) 2 _col0 (type: string) - Statistics: Num rows: 255 Data size: 25572 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 528 Data size: 255728 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false GlobalTableId: 0 @@ -2271,7 +2271,7 @@ STAGE PLANS: Warning: Map Join MAPJOIN[32][bigTable=?] in task 'Stage-8:MAPRED' is a cross product Warning: Map Join MAPJOIN[31][bigTable=?] in task 'Stage-7:MAPRED' is a cross product -Warning: Shuffle Join JOIN[17][tables = [$hdt$_1, $hdt$_2, $hdt$_0, $hdt$_3]] in Stage 'Stage-2:MAPRED' is a cross product +Warning: Shuffle Join JOIN[17][tables = [$hdt$_0, $hdt$_1, $hdt$_2, $hdt$_3]] in Stage 'Stage-2:MAPRED' is a cross product PREHOOK: query: select count(*) FROM bucket_small a JOIN bucket_medium b ON a.key = b.key JOIN bucket_big c ON c.key = b.key JOIN bucket_medium d ON c.key = b.key PREHOOK: type: QUERY PREHOOK: Input: default@bucket_big http://git-wip-us.apache.org/repos/asf/hive/blob/d0fa7d55/ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out ---------------------------------------------------------------------- diff --git a/ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out b/ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out index 4c31a69..ed85c3b 100644 --- a/ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out +++ b/ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out @@ -95,16 +95,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: a - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Sorted Merge Bucket Map Join Operator condition map: Inner Join 0 to 1 @@ -353,16 +353,16 @@ STAGE PLANS: $hdt$_1:b TableScan alias: b - Statistics: Num rows: 2 Data size: 226 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4 Data size: 2260 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 2 Data size: 226 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4 Data size: 2260 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 2 Data size: 226 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4 Data size: 2260 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) @@ -374,16 +374,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: a - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -690,16 +690,16 @@ STAGE PLANS: $hdt$_0:a TableScan alias: a - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) @@ -711,16 +711,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: b - Statistics: Num rows: 2 Data size: 226 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4 Data size: 2260 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 2 Data size: 226 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4 Data size: 2260 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 2 Data size: 226 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4 Data size: 2260 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -924,16 +924,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: a - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Sorted Merge Bucket Map Join Operator condition map: Inner Join 0 to 1 http://git-wip-us.apache.org/repos/asf/hive/blob/d0fa7d55/ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out ---------------------------------------------------------------------- diff --git a/ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out b/ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out index caa3017..fe46bb5 100644 --- a/ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out +++ b/ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out @@ -95,16 +95,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: b - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Sorted Merge Bucket Map Join Operator condition map: Inner Join 0 to 1 @@ -242,16 +242,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: a - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Sorted Merge Bucket Map Join Operator condition map: Inner Join 0 to 1 @@ -497,16 +497,16 @@ STAGE PLANS: $hdt$_1:b TableScan alias: b - Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4 Data size: 2280 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4 Data size: 2280 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4 Data size: 2280 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) @@ -518,16 +518,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: a - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -784,16 +784,16 @@ STAGE PLANS: $hdt$_0:a TableScan alias: a - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) @@ -805,16 +805,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: b - Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4 Data size: 2280 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4 Data size: 2280 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4 Data size: 2280 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -1018,16 +1018,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: a - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 58 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 120 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Sorted Merge Bucket Map Join Operator condition map: Inner Join 0 to 1 http://git-wip-us.apache.org/repos/asf/hive/blob/d0fa7d55/ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out ---------------------------------------------------------------------- diff --git a/ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out b/ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out index 21a1175..91f2f13 100644 --- a/ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out +++ b/ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out @@ -111,16 +111,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: b - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE Sorted Merge Bucket Map Join Operator condition map: Inner Join 0 to 1 @@ -258,16 +258,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: a - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE Sorted Merge Bucket Map Join Operator condition map: Inner Join 0 to 1 @@ -513,16 +513,16 @@ STAGE PLANS: $hdt$_1:b TableScan alias: b - Statistics: Num rows: 4 Data size: 452 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 4520 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 4 Data size: 452 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 4520 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 4 Data size: 452 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 4520 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) @@ -534,16 +534,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: a - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -800,16 +800,16 @@ STAGE PLANS: $hdt$_0:a TableScan alias: a - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) @@ -821,16 +821,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: b - Statistics: Num rows: 4 Data size: 452 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 4520 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 4 Data size: 452 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 4520 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 4 Data size: 452 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 4520 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -1034,16 +1034,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: a - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 27 Data size: 2750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 56 Data size: 27500 Basic stats: COMPLETE Column stats: NONE Sorted Merge Bucket Map Join Operator condition map: Inner Join 0 to 1 http://git-wip-us.apache.org/repos/asf/hive/blob/d0fa7d55/ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out ---------------------------------------------------------------------- diff --git a/ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out b/ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out index a4cdbec..c099242 100644 --- a/ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out +++ b/ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out @@ -128,16 +128,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: b - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Sorted Merge Bucket Map Join Operator condition map: Inner Join 0 to 1 @@ -327,16 +327,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: a - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Sorted Merge Bucket Map Join Operator condition map: Inner Join 0 to 1 @@ -634,16 +634,16 @@ STAGE PLANS: $hdt$_1:b TableScan alias: b - Statistics: Num rows: 4 Data size: 452 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 4520 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 4 Data size: 452 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 4520 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 4 Data size: 452 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 4520 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) @@ -655,16 +655,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: a - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -1019,16 +1019,16 @@ STAGE PLANS: $hdt$_0:a TableScan alias: a - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: string) @@ -1040,16 +1040,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: b - Statistics: Num rows: 4 Data size: 452 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 4520 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 4 Data size: 452 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 4520 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 4 Data size: 452 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 4520 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -1302,16 +1302,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: a - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: string) outputColumnNames: _col0 - Statistics: Num rows: 54 Data size: 5500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 112 Data size: 55000 Basic stats: COMPLETE Column stats: NONE Sorted Merge Bucket Map Join Operator condition map: Inner Join 0 to 1 http://git-wip-us.apache.org/repos/asf/hive/blob/d0fa7d55/ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out ---------------------------------------------------------------------- diff --git a/ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out b/ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out index 7626a73..b891507 100644 --- a/ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out +++ b/ql/src/test/results/clientpositive/avro_schema_evolution_native.q.out @@ -248,11 +248,11 @@ STAGE PLANS: Processor Tree: TableScan alias: episodes_partitioned - Statistics: Num rows: 3 Data size: 889 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 8890 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: title (type: string), air_date (type: string), doctor (type: int), value (type: int), doctor_pt (type: int) outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 3 Data size: 889 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 8890 Basic stats: COMPLETE Column stats: NONE ListSink PREHOOK: query: SELECT * FROM episodes_partitioned WHERE doctor_pt > 6 @@ -337,14 +337,14 @@ STAGE PLANS: Map Operator Tree: TableScan alias: episodes_partitioned - Statistics: Num rows: 3 Data size: 889 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 8890 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: title (type: string), air_date (type: string), doctor (type: int), value (type: int), doctor_pt (type: int) outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 3 Data size: 889 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 8890 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 3 Data size: 889 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3 Data size: 8890 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat http://git-wip-us.apache.org/repos/asf/hive/blob/d0fa7d55/ql/src/test/results/clientpositive/beeline/smb_mapjoin_10.q.out ---------------------------------------------------------------------- diff --git a/ql/src/test/results/clientpositive/beeline/smb_mapjoin_10.q.out b/ql/src/test/results/clientpositive/beeline/smb_mapjoin_10.q.out index 379906e..6f1ba4f 100644 --- a/ql/src/test/results/clientpositive/beeline/smb_mapjoin_10.q.out +++ b/ql/src/test/results/clientpositive/beeline/smb_mapjoin_10.q.out @@ -78,10 +78,10 @@ STAGE PLANS: Map Operator Tree: TableScan alias: b - Statistics: Num rows: 3 Data size: 414 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 13 Data size: 4140 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (pageid is not null and postid is not null and type is not null and userid is not null) (type: boolean) - Statistics: Num rows: 3 Data size: 414 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 13 Data size: 4140 Basic stats: COMPLETE Column stats: NONE Sorted Merge Bucket Map Join Operator condition map: Inner Join 0 to 1 http://git-wip-us.apache.org/repos/asf/hive/blob/d0fa7d55/ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out ---------------------------------------------------------------------- diff --git a/ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out b/ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out index 3545f86..35f05d5 100644 --- a/ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out +++ b/ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out @@ -182,16 +182,16 @@ STAGE PLANS: $hdt$_0:a TableScan alias: a - Statistics: Num rows: 55 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 149 Data size: 58120 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 55 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 149 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) outputColumnNames: _col0, _col1 - Statistics: Num rows: 55 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 149 Data size: 58120 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: int) @@ -203,16 +203,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: b - Statistics: Num rows: 55 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 149 Data size: 58120 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 55 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 149 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) outputColumnNames: _col0, _col1 - Statistics: Num rows: 55 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 149 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -221,17 +221,17 @@ STAGE PLANS: 1 _col0 (type: int) outputColumnNames: _col0, _col1, _col3 Position of Big Table: 1 - Statistics: Num rows: 60 Data size: 6393 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 163 Data size: 63932 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: UDFToString(_col0) (type: string), _col1 (type: string), _col3 (type: string) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 60 Data size: 6393 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 163 Data size: 63932 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false GlobalTableId: 1 #### A masked pattern was here #### NumFilesPerFileSink: 1 - Statistics: Num rows: 60 Data size: 6393 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 163 Data size: 63932 Basic stats: COMPLETE Column stats: NONE #### A masked pattern was here #### table: input format: org.apache.hadoop.mapred.TextInputFormat @@ -261,7 +261,7 @@ STAGE PLANS: Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string) outputColumnNames: key, value1, value2 - Statistics: Num rows: 60 Data size: 6393 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 163 Data size: 63932 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: compute_stats(key, 'hll'), compute_stats(value1, 'hll'), compute_stats(value2, 'hll') mode: hash @@ -619,16 +619,16 @@ STAGE PLANS: $hdt$_0:a TableScan alias: a - Statistics: Num rows: 55 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 149 Data size: 58120 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 55 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 149 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) outputColumnNames: _col0, _col1 - Statistics: Num rows: 55 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 149 Data size: 58120 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: int) @@ -640,16 +640,16 @@ STAGE PLANS: Map Operator Tree: TableScan alias: b - Statistics: Num rows: 55 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 149 Data size: 58120 Basic stats: COMPLETE Column stats: NONE GatherStats: false Filter Operator isSamplingPred: false predicate: key is not null (type: boolean) - Statistics: Num rows: 55 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 149 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) outputColumnNames: _col0, _col1 - Statistics: Num rows: 55 Data size: 5812 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 149 Data size: 58120 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -658,17 +658,17 @@ STAGE PLANS: 1 _col0 (type: int) outputColumnNames: _col0, _col1, _col3 Position of Big Table: 1 - Statistics: Num rows: 60 Data size: 6393 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 163 Data size: 63932 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: UDFToString(_col0) (type: string), _col1 (type: string), _col3 (type: string) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 60 Data size: 6393 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 163 Data size: 63932 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false GlobalTableId: 1 #### A masked pattern was here #### NumFilesPerFileSink: 1 - Statistics: Num rows: 60 Data size: 6393 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 163 Data size: 63932 Basic stats: COMPLETE Column stats: NONE #### A masked pattern was here #### table: input format: org.apache.hadoop.mapred.TextInputFormat @@ -698,7 +698,7 @@ STAGE PLANS: Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string) outputColumnNames: key, value1, value2 - Statistics: Num rows: 60 Data size: 6393 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 163 Data size: 63932 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: compute_stats(key, 'hll'), compute_stats(value1, 'hll'), compute_stats(value2, 'hll') mode: hash