hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Heng Chen <heng.chen.1...@gmail.com>
Subject Re: Could i use Hive SQL parser in our application?
Date Thu, 19 May 2016 06:31:50 GMT
We should use pd.parse(sql) instead of pd.parse(sql, context);

And when i use pd.parse(sql), i could see exception thrown out like below,
it is due to HIVE-10731 <https://issues.apache.org/jira/browse/HIVE-10731>,
 the issue was fixed in 1.3.0,  but i use hive-exec-1.2.0.jar in my
 application.

java.lang.NullPointerException
at org.apache.hadoop.hive.conf.HiveConf.getBoolVar(HiveConf.java:2583)
at
org.apache.hadoop.hive.ql.parse.HiveParser.useSQL11ReservedKeywordsForIdentifier(HiveParser.java:1000)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.useSQL11ReservedKeywordsForIdentifier(HiveParser_IdentifiersParser.java:726)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.functionName(HiveParser_IdentifiersParser.java:4143)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.synpred11_IdentifiersParser_fragment(HiveParser_IdentifiersParser.java:12082)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.synpred11_IdentifiersParser(HiveParser_IdentifiersParser.java:12180)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser$DFA32.specialStateTransition(HiveParser_IdentifiersParser.java)
at org.antlr.runtime.DFA.predict(DFA.java:80)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6177)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
at
org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectExpression(HiveParser_SelectClauseParser.java:3755)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectExpression(HiveParser.java:45854)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.function(HiveParser_IdentifiersParser.java:3850)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6280)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
at
org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectExpression(HiveParser_SelectClauseParser.java:3755)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectExpression(HiveParser.java:45854)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.function(HiveParser_IdentifiersParser.java:3878)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6280)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
at
org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1401)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495)
at
org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:5307)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3741)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1910)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518)
at
org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45873)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516)
at
org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
at
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
at
org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:161)
at
org.apache.hadoop.hive.ql.tools.LineageInfo.getLineageInfo(LineageInfo.java:113)
at
com.fenbi.pipe.utils.LineageInfoUtils.getInputTables(LineageInfoUtils.java:19)
at com.fenbi.pipe.job.tool.ParseSQLTool.run(ParseSQLTool.java:45)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
com.fenbi.pipe.job.hadoop.AbstractPipeJob.runJob(AbstractPipeJob.java:109)
at com.fenbi.pipe.job.tool.ParseSQLTool.main(ParseSQLTool.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)




After i upgrade my hive-exec.jar to 2.0.0,  the above exception disappear,
but it introduce another exception like below by some sql.  Maybe it is
also a bug like HIVE-10731


java.lang.NullPointerException
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:2605)
at
org.apache.hadoop.hive.ql.parse.HiveLexer.allowQuotedId(HiveLexer.java:360)
at
org.apache.hadoop.hive.ql.parse.HiveLexer$DFA21.specialStateTransition(HiveLexer.java:11522)
at org.antlr.runtime.DFA.predict(DFA.java:80)
at
org.apache.hadoop.hive.ql.parse.HiveLexer.mIdentifier(HiveLexer.java:8357)
at org.apache.hadoop.hive.ql.parse.HiveLexer.mTokens(HiveLexer.java:11395)
at org.antlr.runtime.Lexer.nextToken(Lexer.java:89)
at org.antlr.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:133)
at org.antlr.runtime.BufferedTokenStream.sync(BufferedTokenStream.java:127)
at org.antlr.runtime.CommonTokenStream.consume(CommonTokenStream.java:67)
at org.antlr.runtime.DFA.predict(DFA.java:120)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2847)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1429)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1156)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:48008)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:42231)
at
org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:42138)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:41154)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:41024)
at
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1653)
at
org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1137)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:161)
at
org.apache.hadoop.hive.ql.tools.LineageInfo.getLineageInfo(LineageInfo.java:113)
at
com.fenbi.pipe.utils.LineageInfoUtils.getInputTables(LineageInfoUtils.java:19)
at com.fenbi.pipe.job.tool.ParseSQLTool.run(ParseSQLTool.java:45)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
com.fenbi.pipe.job.hadoop.AbstractPipeJob.runJob(AbstractPipeJob.java:109)
at com.fenbi.pipe.job.tool.ParseSQLTool.main(ParseSQLTool.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)



My sql is

select alla.id as questionid,alla.courseid as courseid,alla.grade as
grade,alla.paperid as paperid ,alla.num as num,
alla.type as type,alla.censorstatus as censorstatus,alla.sourcetype as
sourcetype,alla.spidersourcekey as spidersourcekey,t2.toid as
toid,t2.`deleted` as `deleted`,
CASE when sourcetype='1' THEN '小猿搜题'
       when    sourcetype='2' THEN '猿题库'
       when    sourcetype='3' THEN '网络资源'
    when       sourcetype='4' THEN '答题系统'
      when     sourcetype='5' THEN '合作资源'
    when        sourcetype='6' THEN 'Koba'
    when         sourcetype='7' THEN '纸质书'
     when         sourcetype='8' THEN '框图'
     when          sourcetype='9' THEN 'PUZZLE' else 'unknown'
     end as sourcefrom,
    CASE when length(alla.answer)+length(alla.solution)<100  THEN '没有解析'
        else '有解析'
     end as judgesolution,
     CASE when t2.toid is null and t2.`deleted`=0 THEN '不是主原题'
        else '是主原题'
     end as judgeform
from
(
select aq.* ,top.num from
(select count(*) as num, result from dw_solar_query  where result > 0 and
dt>='{{last7Day}}' and dt<='{{yesterday}}'
group by result order by num desc limit 100 ) top
left join apolo_question aq on top.result=aq.id
) alla
left join
(select fromid,`deleted`,toid from sqoop_solar_apolo_question_redirect ) t2
on alla.id=t2.fromid




2016-05-19 12:43 GMT+08:00 Markovitz, Dudu <dmarkovitz@paypal.com>:

> Hi
>
>
>
> Can you please share what was the problem?
>
>
>
> Thanks
>
>
>
> Dudu
>
>
>
> *From:* Heng Chen [mailto:heng.chen.1986@gmail.com]
> *Sent:* Thursday, May 19, 2016 7:07 AM
> *To:* user@hive.apache.org
> *Subject:* Re: Could i use Hive SQL parser in our application?
>
>
>
> Got it now!  Thanks again for your help! guys!
>
>
>
> 2016-05-19 11:09 GMT+08:00 Heng Chen <heng.chen.1986@gmail.com>:
>
> Hi, guys.
>
>
>
>         I write one example as @furcy said like this.
>
>
>
> *public static void *main(String[] args) *throws *SemanticException, ParseException,
IOException {
>   String sql = *"select * from table1 where a > 100"*;
>   Context context = *new *Context(*new *HiveConf());
>   ParseDriver pd = *new *ParseDriver();
>   ASTNode tree = pd.parse(sql, context);
>   System.*out*.println(tree);
> }
>
>
>
> When i run it,  exception thrown out,  did i miss something?
>
>
>
>
>
> Exception in thread "main" java.lang.NullPointerException: Conf non-local
> session path expected to be non-null
>
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
>
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
>
> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:133)
>
> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:120)
>
> at com.fenbi.pipe.utils.LineageInfoUtils.main(LineageInfoUtils.java:24)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:497)
>
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> 2016-05-19 10:20 GMT+08:00 Heng Chen <heng.chen.1986@gmail.com>:
>
> Thanks guys!  Let me try it firstly.
>
>
>
> 2016-05-19 1:44 GMT+08:00 Pengcheng Xiong <pxiong@apache.org>:
>
> Hi Heng,
>
>
>
>     Sure you can. Hive SQL parser is based on ANTLR and you can do that by
> taking that part out of Hive and integrate in to your application. Please
> let me know if you need any further help. Thanks.
>
>
>
> Best
>
> Pengcheng Xiong
>
>
>
> On Wed, May 18, 2016 at 3:43 AM, Heng Chen <heng.chen.1986@gmail.com>
> wrote:
>
> Hi, guys.
>
>
>
>       Recently,  we need to integrate Hive SQL parser in our application.
> Is there any way to do it?
>
>
>
> Thanks!
>
>
>
>
>
>
>
>
>

Mime
View raw message