Return-Path: X-Original-To: apmail-hive-commits-archive@www.apache.org Delivered-To: apmail-hive-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EABAA7D51 for ; Tue, 8 Nov 2011 02:19:45 +0000 (UTC) Received: (qmail 50756 invoked by uid 500); 8 Nov 2011 02:19:45 -0000 Delivered-To: apmail-hive-commits-archive@hive.apache.org Received: (qmail 50730 invoked by uid 500); 8 Nov 2011 02:19:45 -0000 Mailing-List: contact commits-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hive-dev@hive.apache.org Delivered-To: mailing list commits@hive.apache.org Received: (qmail 50720 invoked by uid 99); 8 Nov 2011 02:19:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Nov 2011 02:19:45 +0000 X-ASF-Spam-Status: No, hits=-1998.2 required=5.0 tests=ALL_TRUSTED,LONGWORDS X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Nov 2011 02:19:42 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id D244D238889B for ; Tue, 8 Nov 2011 02:19:20 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1199066 - in /hive/trunk/ql/src: java/org/apache/hadoop/hive/ql/parse/ test/queries/clientpositive/ test/results/clientpositive/ Date: Tue, 08 Nov 2011 02:19:20 -0000 To: commits@hive.apache.org From: jvs@apache.org X-Mailer: svnmailer-1.0.8-patched Message-Id: <20111108021920.D244D238889B@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: jvs Date: Tue Nov 8 02:19:20 2011 New Revision: 1199066 URL: http://svn.apache.org/viewvc?rev=1199066&view=rev Log: HIVE-2527 [jira] Consecutive string literals should be combined into a single string literal. (Jonathan Chang via jvs) Summary: HIVE C, Python, etc. all support this magical feature. Test Plan: EMPTY Reviewers: JIRA, jsichi Reviewed By: jsichi CC: akramer, jonchang, jsichi Differential Revision: 147 Added: hive/trunk/ql/src/test/queries/clientpositive/literal_string.q hive/trunk/ql/src/test/results/clientpositive/literal_string.q.out Modified: hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java Modified: hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g URL: http://svn.apache.org/viewvc/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g?rev=1199066&r1=1199065&r2=1199066&view=diff ============================================================================== --- hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g (original) +++ hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g Tue Nov 8 02:19:20 2011 @@ -181,6 +181,7 @@ TOK_TABLESPLITSAMPLE; TOK_TMP_FILE; TOK_TABSORTCOLNAMEASC; TOK_TABSORTCOLNAMEDESC; +TOK_STRINGLITERALSEQUENCE; TOK_CHARSETLITERAL; TOK_CREATEFUNCTION; TOK_DROPFUNCTION; @@ -1856,6 +1857,7 @@ constant : Number | StringLiteral + | stringLiteralSequence | BigintLiteral | SmallintLiteral | TinyintLiteral @@ -1863,6 +1865,11 @@ constant | booleanValue ; +stringLiteralSequence + : + StringLiteral StringLiteral+ -> ^(TOK_STRINGLITERALSEQUENCE StringLiteral StringLiteral+) + ; + charSetStringLiteral @init { msgs.push("character string literal"); } @after { msgs.pop(); } Modified: hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java URL: http://svn.apache.org/viewvc/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java?rev=1199066&r1=1199065&r2=1199066&view=diff ============================================================================== --- hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (original) +++ hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java Tue Nov 8 02:19:20 2011 @@ -1167,6 +1167,7 @@ public class SemanticAnalyzer extends Ba break; case HiveParser.Number: case HiveParser.StringLiteral: + case HiveParser.TOK_STRINGLITERALSEQUENCE: case HiveParser.TOK_CHARSETLITERAL: case HiveParser.KW_TRUE: case HiveParser.KW_FALSE: Modified: hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java URL: http://svn.apache.org/viewvc/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java?rev=1199066&r1=1199065&r2=1199066&view=diff ============================================================================== --- hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java (original) +++ hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java Tue Nov 8 02:19:20 2011 @@ -136,7 +136,8 @@ public final class TypeCheckProcFactory getNumExprProcessor()); opRules .put(new RuleRegExp("R3", HiveParser.Identifier + "%|" - + HiveParser.StringLiteral + "%|" + HiveParser.TOK_CHARSETLITERAL + + HiveParser.StringLiteral + "%|" + HiveParser.TOK_CHARSETLITERAL + "%|" + + HiveParser.TOK_STRINGLITERALSEQUENCE + "%|" + "%|" + HiveParser.KW_IF + "%|" + HiveParser.KW_CASE + "%|" + HiveParser.KW_WHEN + "%|" + HiveParser.KW_IN + "%|" + HiveParser.KW_ARRAY + "%|" + HiveParser.KW_MAP + "%|" @@ -284,6 +285,14 @@ public final class TypeCheckProcFactory case HiveParser.StringLiteral: str = BaseSemanticAnalyzer.unescapeSQLString(expr.getText()); break; + case HiveParser.TOK_STRINGLITERALSEQUENCE: + StringBuilder sb = new StringBuilder(); + for (Node n : expr.getChildren()) { + sb.append( + BaseSemanticAnalyzer.unescapeSQLString(((ASTNode)n).getText())); + } + str = sb.toString(); + break; case HiveParser.TOK_CHARSETLITERAL: str = BaseSemanticAnalyzer.charSetString(expr.getChild(0).getText(), expr.getChild(1).getText()); Added: hive/trunk/ql/src/test/queries/clientpositive/literal_string.q URL: http://svn.apache.org/viewvc/hive/trunk/ql/src/test/queries/clientpositive/literal_string.q?rev=1199066&view=auto ============================================================================== --- hive/trunk/ql/src/test/queries/clientpositive/literal_string.q (added) +++ hive/trunk/ql/src/test/queries/clientpositive/literal_string.q Tue Nov 8 02:19:20 2011 @@ -0,0 +1,13 @@ +EXPLAIN SELECT 'face''book', 'face' 'book', 'face' + 'book', + "face""book", "face" "book", "face" + "book", + 'face' 'bo' 'ok', 'face'"book", + "face"'book', 'facebook' FROM src LIMIT 1; + +SELECT 'face''book', 'face' 'book', 'face' + 'book', + "face""book", "face" "book", "face" + "book", + 'face' 'bo' 'ok', 'face'"book", + "face"'book', 'facebook' FROM src LIMIT 1; Added: hive/trunk/ql/src/test/results/clientpositive/literal_string.q.out URL: http://svn.apache.org/viewvc/hive/trunk/ql/src/test/results/clientpositive/literal_string.q.out?rev=1199066&view=auto ============================================================================== --- hive/trunk/ql/src/test/results/clientpositive/literal_string.q.out (added) +++ hive/trunk/ql/src/test/results/clientpositive/literal_string.q.out Tue Nov 8 02:19:20 2011 @@ -0,0 +1,83 @@ +PREHOOK: query: EXPLAIN SELECT 'face''book', 'face' 'book', 'face' + 'book', + "face""book", "face" "book", "face" + "book", + 'face' 'bo' 'ok', 'face'"book", + "face"'book', 'facebook' FROM src LIMIT 1 +PREHOOK: type: QUERY +POSTHOOK: query: EXPLAIN SELECT 'face''book', 'face' 'book', 'face' + 'book', + "face""book", "face" "book", "face" + "book", + 'face' 'bo' 'ok', 'face'"book", + "face"'book', 'facebook' FROM src LIMIT 1 +POSTHOOK: type: QUERY +ABSTRACT SYNTAX TREE: + (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME src))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR 'face''book') (TOK_SELEXPR (TOK_STRINGLITERALSEQUENCE 'face' 'book')) (TOK_SELEXPR (TOK_STRINGLITERALSEQUENCE 'face' 'book')) (TOK_SELEXPR "face""book") (TOK_SELEXPR (TOK_STRINGLITERALSEQUENCE "face" "book")) (TOK_SELEXPR (TOK_STRINGLITERALSEQUENCE "face" "book")) (TOK_SELEXPR (TOK_STRINGLITERALSEQUENCE 'face' 'bo' 'ok')) (TOK_SELEXPR 'face'"book") (TOK_SELEXPR "face"'book') (TOK_SELEXPR 'facebook')) (TOK_LIMIT 1))) + +STAGE DEPENDENCIES: + Stage-1 is a root stage + Stage-0 is a root stage + +STAGE PLANS: + Stage: Stage-1 + Map Reduce + Alias -> Map Operator Tree: + src + TableScan + alias: src + Select Operator + expressions: + expr: 'facebook' + type: string + expr: 'facebook' + type: string + expr: 'facebook' + type: string + expr: 'facebook' + type: string + expr: 'facebook' + type: string + expr: 'facebook' + type: string + expr: 'facebook' + type: string + expr: 'facebook' + type: string + expr: 'facebook' + type: string + expr: 'facebook' + type: string + outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9 + Limit + File Output Operator + compressed: false + GlobalTableId: 0 + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + + Stage: Stage-0 + Fetch Operator + limit: 1 + + +PREHOOK: query: SELECT 'face''book', 'face' 'book', 'face' + 'book', + "face""book", "face" "book", "face" + "book", + 'face' 'bo' 'ok', 'face'"book", + "face"'book', 'facebook' FROM src LIMIT 1 +PREHOOK: type: QUERY +PREHOOK: Input: default@src +PREHOOK: Output: file:/var/folders/71/h_j6fpg10r33hvx1lcxlgttcw61_4s/T/jonchang/hive_2011-11-06_11-17-00_593_8968392215771784039/-mr-10000 +POSTHOOK: query: SELECT 'face''book', 'face' 'book', 'face' + 'book', + "face""book", "face" "book", "face" + "book", + 'face' 'bo' 'ok', 'face'"book", + "face"'book', 'facebook' FROM src LIMIT 1 +POSTHOOK: type: QUERY +POSTHOOK: Input: default@src +POSTHOOK: Output: file:/var/folders/71/h_j6fpg10r33hvx1lcxlgttcw61_4s/T/jonchang/hive_2011-11-06_11-17-00_593_8968392215771784039/-mr-10000 +facebook facebook facebook facebook facebook facebook facebook facebook facebook facebook