Return-Path: Delivered-To: apmail-hadoop-hive-commits-archive@minotaur.apache.org Received: (qmail 51913 invoked from network); 16 Sep 2010 16:53:28 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 16 Sep 2010 16:53:28 -0000 Received: (qmail 84792 invoked by uid 500); 16 Sep 2010 16:53:28 -0000 Delivered-To: apmail-hadoop-hive-commits-archive@hadoop.apache.org Received: (qmail 84720 invoked by uid 500); 16 Sep 2010 16:53:26 -0000 Mailing-List: contact hive-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hive-dev@hadoop.apache.org Delivered-To: mailing list hive-commits@hadoop.apache.org Received: (qmail 84712 invoked by uid 99); 16 Sep 2010 16:53:25 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Sep 2010 16:53:25 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Sep 2010 16:53:23 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 19C8723889E0; Thu, 16 Sep 2010 16:53:03 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r997842 - in /hadoop/hive/trunk: ./ ql/src/java/org/apache/hadoop/hive/ql/exec/ ql/src/test/queries/clientpositive/ ql/src/test/results/clientpositive/ Date: Thu, 16 Sep 2010 16:53:02 -0000 To: hive-commits@hadoop.apache.org From: namit@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20100916165303.19C8723889E0@eris.apache.org> Author: namit Date: Thu Sep 16 16:53:02 2010 New Revision: 997842 URL: http://svn.apache.org/viewvc?rev=997842&view=rev Log: HIVE-1639 ExecDriver.addInputPaths() error if partition name contains a comma (Ning Zhang via namit) Modified: hadoop/hive/trunk/CHANGES.txt hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java hadoop/hive/trunk/ql/src/test/queries/clientpositive/merge4.q hadoop/hive/trunk/ql/src/test/results/clientpositive/merge4.q.out hadoop/hive/trunk/ql/src/test/results/clientpositive/merge4.q.out_0.17 Modified: hadoop/hive/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/hive/trunk/CHANGES.txt?rev=997842&r1=997841&r2=997842&view=diff ============================================================================== --- hadoop/hive/trunk/CHANGES.txt (original) +++ hadoop/hive/trunk/CHANGES.txt Thu Sep 16 16:53:02 2010 @@ -253,6 +253,9 @@ Trunk - Unreleased HIVE-1622. Bug in updating log file for hadoop 17 (Ning Zhang via namit) + HIVE-1639 ExecDriver.addInputPaths() error if partition name contains a comma + (Ning Zhang via namit) + TESTS HIVE-1464. improve test query performance Modified: hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java URL: http://svn.apache.org/viewvc/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java?rev=997842&r1=997841&r2=997842&view=diff ============================================================================== --- hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java (original) +++ hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java Thu Sep 16 16:53:02 2010 @@ -465,12 +465,12 @@ public class ExecDriver extends Task Map Operator Tree: - pfile:/data/users/nzhang/work/870/apache-hive/build/ql/scratchdir/hive_2010-08-18_23-49-57_428_7822263291550287040/-ext-10002 + pfile:/data/users/nzhang/work/870/apache-hive/build/ql/scratchdir/hive_2010-09-15_16-43-51_321_4342271261883953579/-ext-10002 File Output Operator compressed: false GlobalTableId: 0 @@ -106,12 +106,12 @@ PREHOOK: query: select * from nzhang_par PREHOOK: type: QUERY PREHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=11 PREHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=12 -PREHOOK: Output: file:/tmp/nzhang/hive_2010-08-18_23-50-08_422_6318944484933266874/-mr-10000 +PREHOOK: Output: file:/tmp/nzhang/hive_2010-09-15_16-44-02_504_6924515535764392544/-mr-10000 POSTHOOK: query: select * from nzhang_part POSTHOOK: type: QUERY POSTHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=11 POSTHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=12 -POSTHOOK: Output: file:/tmp/nzhang/hive_2010-08-18_23-50-08_422_6318944484933266874/-mr-10000 +POSTHOOK: Output: file:/tmp/nzhang/hive_2010-09-15_16-44-02_504_6924515535764392544/-mr-10000 POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] @@ -1174,7 +1174,7 @@ STAGE PLANS: Move Operator files: hdfs directory: true - destination: pfile:/data/users/nzhang/work/870/apache-hive/build/ql/scratchdir/hive_2010-08-18_23-50-09_351_726334652087316579/-ext-10000 + destination: pfile:/data/users/nzhang/work/870/apache-hive/build/ql/scratchdir/hive_2010-09-15_16-44-03_589_2316123978849728483/-ext-10000 Stage: Stage-0 Move Operator @@ -1192,7 +1192,7 @@ STAGE PLANS: Stage: Stage-2 Map Reduce Alias -> Map Operator Tree: - pfile:/data/users/nzhang/work/870/apache-hive/build/ql/scratchdir/hive_2010-08-18_23-50-09_351_726334652087316579/-ext-10002 + pfile:/data/users/nzhang/work/870/apache-hive/build/ql/scratchdir/hive_2010-09-15_16-44-03_589_2316123978849728483/-ext-10002 File Output Operator compressed: false GlobalTableId: 0 @@ -1223,12 +1223,12 @@ PREHOOK: query: select * from nzhang_par PREHOOK: type: QUERY PREHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=11 PREHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=12 -PREHOOK: Output: file:/tmp/nzhang/hive_2010-08-18_23-50-19_197_960198185173825453/-mr-10000 +PREHOOK: Output: file:/tmp/nzhang/hive_2010-09-15_16-44-13_644_4054388057551943925/-mr-10000 POSTHOOK: query: select * from nzhang_part POSTHOOK: type: QUERY POSTHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=11 POSTHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=12 -POSTHOOK: Output: file:/tmp/nzhang/hive_2010-08-18_23-50-19_197_960198185173825453/-mr-10000 +POSTHOOK: Output: file:/tmp/nzhang/hive_2010-09-15_16-44-13_644_4054388057551943925/-mr-10000 POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] @@ -2735,3 +2735,253 @@ POSTHOOK: Lineage: nzhang_part PARTITION 400 val_400 2010-08-15 12 200 val_200 2010-08-15 12 97 val_97 2010-08-15 12 +PREHOOK: query: explain +insert overwrite table nzhang_part partition (ds='2010-08-15', hr) +select * from ( + select key, value, hr from srcpart where ds='2008-04-08' + union all + select '1' as key, '1' as value, 'file,' as hr from src limit 1) s +PREHOOK: type: QUERY +POSTHOOK: query: explain +insert overwrite table nzhang_part partition (ds='2010-08-15', hr) +select * from ( + select key, value, hr from srcpart where ds='2008-04-08' + union all + select '1' as key, '1' as value, 'file,' as hr from src limit 1) s +POSTHOOK: type: QUERY +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +ABSTRACT SYNTAX TREE: + (TOK_QUERY (TOK_FROM (TOK_SUBQUERY (TOK_UNION (TOK_QUERY (TOK_FROM (TOK_TABREF srcpart)) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL key)) (TOK_SELEXPR (TOK_TABLE_OR_COL value)) (TOK_SELEXPR (TOK_TABLE_OR_COL hr))) (TOK_WHERE (= (TOK_TABLE_OR_COL ds) '2008-04-08')))) (TOK_QUERY (TOK_FROM (TOK_TABREF src)) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR '1' key) (TOK_SELEXPR '1' value) (TOK_SELEXPR 'file,' hr)) (TOK_LIMIT 1)))) s)) (TOK_INSERT (TOK_DESTINATION (TOK_TAB nzhang_part (TOK_PARTSPEC (TOK_PARTVAL ds '2010-08-15') (TOK_PARTVAL hr)))) (TOK_SELECT (TOK_SELEXPR TOK_ALLCOLREF)))) + +STAGE DEPENDENCIES: + Stage-1 is a root stage + Stage-2 depends on stages: Stage-1, Stage-6 + Stage-5 depends on stages: Stage-2 , consists of Stage-4, Stage-3 + Stage-4 + Stage-0 depends on stages: Stage-4, Stage-3 + Stage-3 + Stage-6 is a root stage + +STAGE PLANS: + Stage: Stage-1 + Map Reduce + Alias -> Map Operator Tree: + null-subquery1:s-subquery1:srcpart + TableScan + alias: srcpart + Filter Operator + predicate: + expr: (ds = '2008-04-08') + type: boolean + Filter Operator + predicate: + expr: (ds = '2008-04-08') + type: boolean + Select Operator + expressions: + expr: key + type: string + expr: value + type: string + expr: hr + type: string + outputColumnNames: _col0, _col1, _col2 + File Output Operator + compressed: false + GlobalTableId: 0 + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + + Stage: Stage-2 + Map Reduce + Alias -> Map Operator Tree: + file:/tmp/nzhang/hive_2010-09-15_16-44-14_463_7249270881536354004/-mr-10002 + Union + Select Operator + expressions: + expr: _col0 + type: string + expr: _col1 + type: string + expr: _col2 + type: string + outputColumnNames: _col0, _col1, _col2 + File Output Operator + compressed: false + GlobalTableId: 1 + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + name: nzhang_part + file:/tmp/nzhang/hive_2010-09-15_16-44-14_463_7249270881536354004/-mr-10004 + Union + Select Operator + expressions: + expr: _col0 + type: string + expr: _col1 + type: string + expr: _col2 + type: string + outputColumnNames: _col0, _col1, _col2 + File Output Operator + compressed: false + GlobalTableId: 1 + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + name: nzhang_part + + Stage: Stage-5 + Conditional Operator + + Stage: Stage-4 + Move Operator + files: + hdfs directory: true + destination: pfile:/data/users/nzhang/work/870/apache-hive/build/ql/scratchdir/hive_2010-09-15_16-44-14_463_7249270881536354004/-ext-10000 + + Stage: Stage-0 + Move Operator + tables: + partition: + ds 2010-08-15 + hr + replace: true + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + name: nzhang_part + + Stage: Stage-3 + Map Reduce + Alias -> Map Operator Tree: + pfile:/data/users/nzhang/work/870/apache-hive/build/ql/scratchdir/hive_2010-09-15_16-44-14_463_7249270881536354004/-ext-10003 + File Output Operator + compressed: false + GlobalTableId: 0 + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + name: nzhang_part + + Stage: Stage-6 + Map Reduce + Alias -> Map Operator Tree: + null-subquery2:s-subquery2:src + TableScan + alias: src + Select Operator + expressions: + expr: '1' + type: string + expr: '1' + type: string + expr: 'file,' + type: string + outputColumnNames: _col0, _col1, _col2 + Limit + Reduce Output Operator + sort order: + tag: -1 + value expressions: + expr: _col0 + type: string + expr: _col1 + type: string + expr: _col2 + type: string + Reduce Operator Tree: + Extract + Limit + File Output Operator + compressed: false + GlobalTableId: 0 + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + + +PREHOOK: query: insert overwrite table nzhang_part partition (ds='2010-08-15', hr) +select * from ( + select key, value, hr from srcpart where ds='2008-04-08' + union all + select '1' as key, '1' as value, 'file,' as hr from src limit 1) s +PREHOOK: type: QUERY +PREHOOK: Input: default@src +PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 +PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=12 +POSTHOOK: query: insert overwrite table nzhang_part partition (ds='2010-08-15', hr) +select * from ( + select key, value, hr from srcpart where ds='2008-04-08' + union all + select '1' as key, '1' as value, 'file,' as hr from src limit 1) s +POSTHOOK: type: QUERY +POSTHOOK: Input: default@src +POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 +POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12 +POSTHOOK: Output: default@nzhang_part@ds=2010-08-15/hr=11 +POSTHOOK: Output: default@nzhang_part@ds=2010-08-15/hr=12 +POSTHOOK: Output: default@nzhang_part@ds=2010-08-15/hr=file, +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +PREHOOK: query: show partitions nzhang_part +PREHOOK: type: SHOWPARTITIONS +POSTHOOK: query: show partitions nzhang_part +POSTHOOK: type: SHOWPARTITIONS +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +ds=2010-08-15/hr=11 +ds=2010-08-15/hr=12 +ds=2010-08-15/hr=file, +PREHOOK: query: select * from nzhang_part where hr = 'file,' +PREHOOK: type: QUERY +PREHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=file, +PREHOOK: Output: file:/tmp/nzhang/hive_2010-09-15_16-44-35_568_3765164280240692375/-mr-10000 +POSTHOOK: query: select * from nzhang_part where hr = 'file,' +POSTHOOK: type: QUERY +POSTHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=file, +POSTHOOK: Output: file:/tmp/nzhang/hive_2010-09-15_16-44-35_568_3765164280240692375/-mr-10000 +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +1 1 2010-08-15 file, Modified: hadoop/hive/trunk/ql/src/test/results/clientpositive/merge4.q.out_0.17 URL: http://svn.apache.org/viewvc/hadoop/hive/trunk/ql/src/test/results/clientpositive/merge4.q.out_0.17?rev=997842&r1=997841&r2=997842&view=diff ============================================================================== --- hadoop/hive/trunk/ql/src/test/results/clientpositive/merge4.q.out_0.17 (original) +++ hadoop/hive/trunk/ql/src/test/results/clientpositive/merge4.q.out_0.17 Thu Sep 16 16:53:02 2010 @@ -81,12 +81,12 @@ PREHOOK: query: select * from nzhang_par PREHOOK: type: QUERY PREHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=11 PREHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=12 -PREHOOK: Output: file:/tmp/nzhang/hive_2010-08-19_12-34-01_694_1262392506003674198/-mr-10000 +PREHOOK: Output: file:/tmp/nzhang/hive_2010-09-15_16-54-30_838_7629815615150053920/-mr-10000 POSTHOOK: query: select * from nzhang_part POSTHOOK: type: QUERY POSTHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=11 POSTHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=12 -POSTHOOK: Output: file:/tmp/nzhang/hive_2010-08-19_12-34-01_694_1262392506003674198/-mr-10000 +POSTHOOK: Output: file:/tmp/nzhang/hive_2010-09-15_16-54-30_838_7629815615150053920/-mr-10000 POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] @@ -1173,12 +1173,12 @@ PREHOOK: query: select * from nzhang_par PREHOOK: type: QUERY PREHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=11 PREHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=12 -PREHOOK: Output: file:/tmp/nzhang/hive_2010-08-19_12-34-07_502_6771693234182088332/-mr-10000 +PREHOOK: Output: file:/tmp/nzhang/hive_2010-09-15_16-54-39_689_174314225109504234/-mr-10000 POSTHOOK: query: select * from nzhang_part POSTHOOK: type: QUERY POSTHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=11 POSTHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=12 -POSTHOOK: Output: file:/tmp/nzhang/hive_2010-08-19_12-34-07_502_6771693234182088332/-mr-10000 +POSTHOOK: Output: file:/tmp/nzhang/hive_2010-09-15_16-54-39_689_174314225109504234/-mr-10000 POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] @@ -2685,3 +2685,228 @@ POSTHOOK: Lineage: nzhang_part PARTITION 400 val_400 2010-08-15 12 200 val_200 2010-08-15 12 97 val_97 2010-08-15 12 +PREHOOK: query: explain +insert overwrite table nzhang_part partition (ds='2010-08-15', hr) +select * from ( + select key, value, hr from srcpart where ds='2008-04-08' + union all + select '1' as key, '1' as value, 'file,' as hr from src limit 1) s +PREHOOK: type: QUERY +POSTHOOK: query: explain +insert overwrite table nzhang_part partition (ds='2010-08-15', hr) +select * from ( + select key, value, hr from srcpart where ds='2008-04-08' + union all + select '1' as key, '1' as value, 'file,' as hr from src limit 1) s +POSTHOOK: type: QUERY +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +ABSTRACT SYNTAX TREE: + (TOK_QUERY (TOK_FROM (TOK_SUBQUERY (TOK_UNION (TOK_QUERY (TOK_FROM (TOK_TABREF srcpart)) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL key)) (TOK_SELEXPR (TOK_TABLE_OR_COL value)) (TOK_SELEXPR (TOK_TABLE_OR_COL hr))) (TOK_WHERE (= (TOK_TABLE_OR_COL ds) '2008-04-08')))) (TOK_QUERY (TOK_FROM (TOK_TABREF src)) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR '1' key) (TOK_SELEXPR '1' value) (TOK_SELEXPR 'file,' hr)) (TOK_LIMIT 1)))) s)) (TOK_INSERT (TOK_DESTINATION (TOK_TAB nzhang_part (TOK_PARTSPEC (TOK_PARTVAL ds '2010-08-15') (TOK_PARTVAL hr)))) (TOK_SELECT (TOK_SELEXPR TOK_ALLCOLREF)))) + +STAGE DEPENDENCIES: + Stage-1 is a root stage + Stage-2 depends on stages: Stage-1, Stage-3 + Stage-0 depends on stages: Stage-2 + Stage-3 is a root stage + +STAGE PLANS: + Stage: Stage-1 + Map Reduce + Alias -> Map Operator Tree: + null-subquery1:s-subquery1:srcpart + TableScan + alias: srcpart + Filter Operator + predicate: + expr: (ds = '2008-04-08') + type: boolean + Filter Operator + predicate: + expr: (ds = '2008-04-08') + type: boolean + Select Operator + expressions: + expr: key + type: string + expr: value + type: string + expr: hr + type: string + outputColumnNames: _col0, _col1, _col2 + File Output Operator + compressed: false + GlobalTableId: 0 + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + + Stage: Stage-2 + Map Reduce + Alias -> Map Operator Tree: + file:/tmp/nzhang/hive_2010-09-15_16-54-41_064_902801052919283411/-mr-10002 + Union + Select Operator + expressions: + expr: _col0 + type: string + expr: _col1 + type: string + expr: _col2 + type: string + outputColumnNames: _col0, _col1, _col2 + File Output Operator + compressed: false + GlobalTableId: 1 + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + name: nzhang_part + file:/tmp/nzhang/hive_2010-09-15_16-54-41_064_902801052919283411/-mr-10003 + Union + Select Operator + expressions: + expr: _col0 + type: string + expr: _col1 + type: string + expr: _col2 + type: string + outputColumnNames: _col0, _col1, _col2 + File Output Operator + compressed: false + GlobalTableId: 1 + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + name: nzhang_part + + Stage: Stage-0 + Move Operator + tables: + partition: + ds 2010-08-15 + hr + replace: true + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + name: nzhang_part + + Stage: Stage-3 + Map Reduce + Alias -> Map Operator Tree: + null-subquery2:s-subquery2:src + TableScan + alias: src + Select Operator + expressions: + expr: '1' + type: string + expr: '1' + type: string + expr: 'file,' + type: string + outputColumnNames: _col0, _col1, _col2 + Limit + Reduce Output Operator + sort order: + tag: -1 + value expressions: + expr: _col0 + type: string + expr: _col1 + type: string + expr: _col2 + type: string + Reduce Operator Tree: + Extract + Limit + File Output Operator + compressed: false + GlobalTableId: 0 + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + + +PREHOOK: query: insert overwrite table nzhang_part partition (ds='2010-08-15', hr) +select * from ( + select key, value, hr from srcpart where ds='2008-04-08' + union all + select '1' as key, '1' as value, 'file,' as hr from src limit 1) s +PREHOOK: type: QUERY +PREHOOK: Input: default@src +PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 +PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=12 +POSTHOOK: query: insert overwrite table nzhang_part partition (ds='2010-08-15', hr) +select * from ( + select key, value, hr from srcpart where ds='2008-04-08' + union all + select '1' as key, '1' as value, 'file,' as hr from src limit 1) s +POSTHOOK: type: QUERY +POSTHOOK: Input: default@src +POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 +POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12 +POSTHOOK: Output: default@nzhang_part@ds=2010-08-15/hr=11 +POSTHOOK: Output: default@nzhang_part@ds=2010-08-15/hr=12 +POSTHOOK: Output: default@nzhang_part@ds=2010-08-15/hr=file, +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +PREHOOK: query: show partitions nzhang_part +PREHOOK: type: SHOWPARTITIONS +POSTHOOK: query: show partitions nzhang_part +POSTHOOK: type: SHOWPARTITIONS +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +ds=2010-08-15/hr=11 +ds=2010-08-15/hr=12 +ds=2010-08-15/hr=file, +PREHOOK: query: select * from nzhang_part where hr = 'file,' +PREHOOK: type: QUERY +PREHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=file, +PREHOOK: Output: file:/tmp/nzhang/hive_2010-09-15_16-54-54_825_7294798270721816998/-mr-10000 +POSTHOOK: query: select * from nzhang_part where hr = 'file,' +POSTHOOK: type: QUERY +POSTHOOK: Input: default@nzhang_part@ds=2010-08-15/hr=file, +POSTHOOK: Output: file:/tmp/nzhang/hive_2010-09-15_16-54-54_825_7294798270721816998/-mr-10000 +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key SIMPLE [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value SIMPLE [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] +POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] +1 1 2010-08-15 file,