hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 刘虓 <ipf...@gmail.com>
Subject Re: Could i use Hive SQL parser in our application?
Date Thu, 19 May 2016 06:43:15 GMT
Hi,
I wrote this hive ql ast visualizer tool last summer using JRuby,hope it
would help.
here:
https://github.com/liprais/hive_ast_tree_vis


2016-05-19 14:31 GMT+08:00 Heng Chen <heng.chen.1986@gmail.com>:

> We should use pd.parse(sql) instead of pd.parse(sql, context);
>
> And when i use pd.parse(sql), i could see exception thrown out like below,
> it is due to HIVE-10731 <https://issues.apache.org/jira/browse/HIVE-10731>,
>  the issue was fixed in 1.3.0,  but i use hive-exec-1.2.0.jar in my
>  application.
>
> java.lang.NullPointerException
> at org.apache.hadoop.hive.conf.HiveConf.getBoolVar(HiveConf.java:2583)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.useSQL11ReservedKeywordsForIdentifier(HiveParser.java:1000)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.useSQL11ReservedKeywordsForIdentifier(HiveParser_IdentifiersParser.java:726)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.functionName(HiveParser_IdentifiersParser.java:4143)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.synpred11_IdentifiersParser_fragment(HiveParser_IdentifiersParser.java:12082)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.synpred11_IdentifiersParser(HiveParser_IdentifiersParser.java:12180)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser$DFA32.specialStateTransition(HiveParser_IdentifiersParser.java)
> at org.antlr.runtime.DFA.predict(DFA.java:80)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6177)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectExpression(HiveParser_SelectClauseParser.java:3755)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectExpression(HiveParser.java:45854)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.function(HiveParser_IdentifiersParser.java:3850)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6280)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectExpression(HiveParser_SelectClauseParser.java:3755)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectExpression(HiveParser.java:45854)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.function(HiveParser_IdentifiersParser.java:3878)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6280)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1401)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:5307)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3741)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1910)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45873)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:161)
> at
> org.apache.hadoop.hive.ql.tools.LineageInfo.getLineageInfo(LineageInfo.java:113)
> at
> com.fenbi.pipe.utils.LineageInfoUtils.getInputTables(LineageInfoUtils.java:19)
> at com.fenbi.pipe.job.tool.ParseSQLTool.run(ParseSQLTool.java:45)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> com.fenbi.pipe.job.hadoop.AbstractPipeJob.runJob(AbstractPipeJob.java:109)
> at com.fenbi.pipe.job.tool.ParseSQLTool.main(ParseSQLTool.java:54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>
>
>
>
> After i upgrade my hive-exec.jar to 2.0.0,  the above exception disappear,
> but it introduce another exception like below by some sql.  Maybe it is
> also a bug like HIVE-10731
>
>
> java.lang.NullPointerException
> at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:2605)
> at
> org.apache.hadoop.hive.ql.parse.HiveLexer.allowQuotedId(HiveLexer.java:360)
> at
> org.apache.hadoop.hive.ql.parse.HiveLexer$DFA21.specialStateTransition(HiveLexer.java:11522)
> at org.antlr.runtime.DFA.predict(DFA.java:80)
> at
> org.apache.hadoop.hive.ql.parse.HiveLexer.mIdentifier(HiveLexer.java:8357)
> at org.apache.hadoop.hive.ql.parse.HiveLexer.mTokens(HiveLexer.java:11395)
> at org.antlr.runtime.Lexer.nextToken(Lexer.java:89)
> at
> org.antlr.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:133)
> at org.antlr.runtime.BufferedTokenStream.sync(BufferedTokenStream.java:127)
> at org.antlr.runtime.CommonTokenStream.consume(CommonTokenStream.java:67)
> at org.antlr.runtime.DFA.predict(DFA.java:120)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2847)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1429)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1156)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:48008)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:42231)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:42138)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:41154)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:41024)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1653)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1137)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:161)
> at
> org.apache.hadoop.hive.ql.tools.LineageInfo.getLineageInfo(LineageInfo.java:113)
> at
> com.fenbi.pipe.utils.LineageInfoUtils.getInputTables(LineageInfoUtils.java:19)
> at com.fenbi.pipe.job.tool.ParseSQLTool.run(ParseSQLTool.java:45)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> com.fenbi.pipe.job.hadoop.AbstractPipeJob.runJob(AbstractPipeJob.java:109)
> at com.fenbi.pipe.job.tool.ParseSQLTool.main(ParseSQLTool.java:54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>
>
>
> My sql is
>
> select alla.id as questionid,alla.courseid as courseid,alla.grade as
> grade,alla.paperid as paperid ,alla.num as num,
> alla.type as type,alla.censorstatus as censorstatus,alla.sourcetype as
> sourcetype,alla.spidersourcekey as spidersourcekey,t2.toid as
> toid,t2.`deleted` as `deleted`,
> CASE when sourcetype='1' THEN '小猿搜题'
>        when    sourcetype='2' THEN '猿题库'
>        when    sourcetype='3' THEN '网络资源'
>     when       sourcetype='4' THEN '答题系统'
>       when     sourcetype='5' THEN '合作资源'
>     when        sourcetype='6' THEN 'Koba'
>     when         sourcetype='7' THEN '纸质书'
>      when         sourcetype='8' THEN '框图'
>      when          sourcetype='9' THEN 'PUZZLE' else 'unknown'
>      end as sourcefrom,
>     CASE when length(alla.answer)+length(alla.solution)<100  THEN '没有解析'
>         else '有解析'
>      end as judgesolution,
>      CASE when t2.toid is null and t2.`deleted`=0 THEN '不是主原题'
>         else '是主原题'
>      end as judgeform
> from
> (
> select aq.* ,top.num from
> (select count(*) as num, result from dw_solar_query  where result > 0 and
> dt>='{{last7Day}}' and dt<='{{yesterday}}'
> group by result order by num desc limit 100 ) top
> left join apolo_question aq on top.result=aq.id
> ) alla
> left join
> (select fromid,`deleted`,toid from sqoop_solar_apolo_question_redirect )
> t2
> on alla.id=t2.fromid
>
>
>
>
> 2016-05-19 12:43 GMT+08:00 Markovitz, Dudu <dmarkovitz@paypal.com>:
>
>> Hi
>>
>>
>>
>> Can you please share what was the problem?
>>
>>
>>
>> Thanks
>>
>>
>>
>> Dudu
>>
>>
>>
>> *From:* Heng Chen [mailto:heng.chen.1986@gmail.com]
>> *Sent:* Thursday, May 19, 2016 7:07 AM
>> *To:* user@hive.apache.org
>> *Subject:* Re: Could i use Hive SQL parser in our application?
>>
>>
>>
>> Got it now!  Thanks again for your help! guys!
>>
>>
>>
>> 2016-05-19 11:09 GMT+08:00 Heng Chen <heng.chen.1986@gmail.com>:
>>
>> Hi, guys.
>>
>>
>>
>>         I write one example as @furcy said like this.
>>
>>
>>
>> *public static void *main(String[] args) *throws *SemanticException, ParseException,
IOException {
>>   String sql = *"select * from table1 where a > 100"*;
>>   Context context = *new *Context(*new *HiveConf());
>>   ParseDriver pd = *new *ParseDriver();
>>   ASTNode tree = pd.parse(sql, context);
>>   System.*out*.println(tree);
>> }
>>
>>
>>
>> When i run it,  exception thrown out,  did i miss something?
>>
>>
>>
>>
>>
>> Exception in thread "main" java.lang.NullPointerException: Conf non-local
>> session path expected to be non-null
>>
>> at
>> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
>>
>> at
>> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
>>
>> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:133)
>>
>> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:120)
>>
>> at com.fenbi.pipe.utils.LineageInfoUtils.main(LineageInfoUtils.java:24)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:497)
>>
>> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 2016-05-19 10:20 GMT+08:00 Heng Chen <heng.chen.1986@gmail.com>:
>>
>> Thanks guys!  Let me try it firstly.
>>
>>
>>
>> 2016-05-19 1:44 GMT+08:00 Pengcheng Xiong <pxiong@apache.org>:
>>
>> Hi Heng,
>>
>>
>>
>>     Sure you can. Hive SQL parser is based on ANTLR and you can do that
>> by taking that part out of Hive and integrate in to your application.
>> Please let me know if you need any further help. Thanks.
>>
>>
>>
>> Best
>>
>> Pengcheng Xiong
>>
>>
>>
>> On Wed, May 18, 2016 at 3:43 AM, Heng Chen <heng.chen.1986@gmail.com>
>> wrote:
>>
>> Hi, guys.
>>
>>
>>
>>       Recently,  we need to integrate Hive SQL parser in our
>> application.  Is there any way to do it?
>>
>>
>>
>> Thanks!
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>

Mime
View raw message