hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pengcheng Xiong (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-8997) Groupby index will fail if an indexed group by operator is followed by a non-indexed group by operator
Date Sat, 29 Nov 2014 07:13:12 GMT
Pengcheng Xiong created HIVE-8997:
-------------------------------------

             Summary: Groupby index will fail if an indexed group by operator is followed
by a non-indexed group by operator
                 Key: HIVE-8997
                 URL: https://issues.apache.org/jira/browse/HIVE-8997
             Project: Hive
          Issue Type: Bug
            Reporter: Pengcheng Xiong
            Assignee: Pengcheng Xiong


following ql_rewrite_gbtoidx.q, if we run

explain
select ckeysum, count(ckeysum)
from
(select l_shipdate, count(l_shipdate) as ckeysum
from lineitem_ix
group by l_shipdate) tabA
group by ckeysum

We will get an error:

junit.framework.AssertionFailedError: Client Execution failed with error code = 40000 running

The trace is 

MismatchedTokenException(-1!=12)
at org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.charSetStringLiteral(HiveParser_IdentifiersParser.java:6099)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.constant(HiveParser_IdentifiersParser.java:5891)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6478)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6641)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:7026)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:7086)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7270)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7430)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7590)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7750)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7909)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8439)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9452)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9571)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9730)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6363)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.groupByExpression(HiveParser_IdentifiersParser.java:1386)
at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.groupByClause(HiveParser_IdentifiersParser.java:774)
at org.apache.hadoop.hive.ql.parse.HiveParser.groupByClause(HiveParser.java:44007)
at org.apache.hadoop.hive.ql.parse.HiveParser.singleSelectStatement(HiveParser.java:41504)
at org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41135)
at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41072)
at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40125)
at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40001)
at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1519)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1057)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.optimizer.index.RewriteParseContextGenerator.generateOperatorTree(RewriteParseContextGenerator.java:67)
at org.apache.hadoop.hive.ql.optimizer.index.RewriteQueryUsingAggregateIndex$NewQueryGroupbySchemaProc.process(RewriteQueryUsingAggregateIndex.java:255)
at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at org.apache.hadoop.hive.ql.optimizer.index.RewriteQueryUsingAggregateIndexCtx.invokeRewriteQueryProc(RewriteQueryUsingAggregateIndexCtx.java:151)
at org.apache.hadoop.hive.ql.optimizer.index.RewriteGBUsingIndex.rewriteOriginalQuery(RewriteGBUsingIndex.java:382)
at org.apache.hadoop.hive.ql.optimizer.index.RewriteGBUsingIndex.transform(RewriteGBUsingIndex.java:135)
at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:177)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10177)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:419)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1107)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1169)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1034)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:199)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:151)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:362)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:297)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:849)
at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:136)
at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx_notwork(TestCliDriver.java:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Exception: Client Execution failed with error code = 40000 running

The main reason is that
After Optimizer,org.apache.hadoop.hive.ql.optimizer.ColumnPruner, we should get the operator
tree as
TS[0]-SEL[1]-GBY[2]-RS[3]-GBY[4]-SEL[5]-SEL[6]-GBY[7]-RS[8]-GBY[9]-SEL[10]-FS[11]

Here, TS[0], SEL[1], GBY[2], GBY[4] should be rewritten. However, because the default graph
walker is used in the code, GBY[9] and GBY[7] are going to be rewritten. This causes the error.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message