hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karan Kumar <karankumar1...@gmail.com>
Subject Re: array<string> become struct<> when doing select
Date Tue, 07 Jul 2015 13:03:40 GMT
Issue is with the thrift version you are using most probably.
https://issues.apache.org/jira/browse/THRIFT-2172

I used thrift-0.9.2 to generate my thrift classes which solved this kind of
issue.

On Tue, Jul 7, 2015 at 6:14 PM, Binglin Chang <decstery@gmail.com> wrote:

> Sorry, forgot to mention, the table is using thrift serde, but 'show
> create table' shows the table is ROW FORMAT DELIMITED, which I think is a
> bug.
> When select simple text format table, the query runs fine, but when select
>  thrift table,  error occurs.
>
> original create table statement:
>
> CREATE EXTERNAL TABLE xxx(commonUserId
> STRUCT<id:STRING,idType:STRING>,lastActiveTime BIGINT,searchWords
> ARRAY<STRING>,calls MAP<STRING,ARRAY<BIGINT>>,hasUsedRecharge
> INT,hasUsedExpress INT,hasUsedViolateRegulation
> INT,hasUsedLicensePlateLottery INT,interestedShops ARRAY<STRING>) ROW
> FORMAT SERDE 'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' WITH
> SERDEPROPERTIES('serialization.format'='org.apache.thrift.protocol.TCompactProtocol','serialization.class'='com.xiaomi.data.spec.platform.xxx')
> STORED AS SEQUENCEFILE LOCATION 'xxxx'
>
>
>
>
> On Tue, Jul 7, 2015 at 7:44 PM, Binglin Chang <decstery@gmail.com> wrote:
>
>> Hi,
>>
>> I have a table with some array fields, when preview them using "select
>> limit" at beeline, I got following errors, it seems the typeinfo string is
>> changed from array<string> to struct<>
>> I am using hive-0.13.1
>>
>> 0: jdbc:hive2://lg-hadoop-hive01.bj:32203/> show create table xxx;
>>
>> +----------------------------------------------------------------------------------------+--+
>> |                                     createtab_stmt
>>                 |
>>
>> +----------------------------------------------------------------------------------------+--+
>> | CREATE EXTERNAL TABLE `xxx`(                                    |
>> |   `commonuserid` struct<id:string,idType:string>,
>>                |
>> |   `lastactivetime` bigint,
>>                 |
>> |   `searchwords` array<string>,
>>                 |
>> |   `calls` map<string,array<bigint>>,
>>                 |
>> |   `hasusedrecharge` int,
>>                 |
>> |   `hasusedexpress` int,
>>                |
>> |   `hasusedviolateregulation` int,
>>                |
>> |   `hasusedlicenseplatelottery` int,
>>                |
>> |   `interestedshops` array<string>)
>>                 |
>> | ROW FORMAT DELIMITED
>>                 |
>> | STORED AS INPUTFORMAT
>>                |
>> |   'org.apache.hadoop.mapred.SequenceFileInputFormat'
>>                 |
>> | OUTPUTFORMAT
>>                 |
>> |   'org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat'
>>                |
>>
>>
>> 0: jdbc:hive2://lg-hadoop-hive01.bj:32203/> select searchwords from
>>  yellowpage.yp_user_actions limit 1;
>> Error: Error while compiling statement: FAILED: SemanticException
>> java.lang.IllegalArgumentException: Error: name expected at the position 7
>> of 'struct<>' but '>' is found. (state=42000,code=40000)
>>
>> Full stack:
>>
>> org.apache.hadoop.hive.ql.parse.SemanticException: java.lang.IllegalArgumentException:
Error: name expected at the position 7 of 'struct<>' but '>' is found.
>> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genConversionSelectOperator(SemanticAnalyzer.java:5949)
>> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:5845)
>> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8235)
>> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8126)
>> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:8956)
>> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9209)
>> 	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:206)
>> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:435)
>> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:333)
>> 	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:989)
>> 	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:982)
>> 	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:123)
>> 	at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:197)
>> 	at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:734)
>> 	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:376)
>> 	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:362)
>> 	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:240)
>> 	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:378)
>> 	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1373)
>> 	at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1358)
>> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>> 	at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:677)
>> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> 	at java.lang.Thread.run(Thread.java:662)
>> Caused by: java.lang.IllegalArgumentException: Error: name expected at the position
7 of 'struct<>' but '>' is found.
>> 	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(TypeInfoUtils.java:354)
>> 	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(TypeInfoUtils.java:331)
>> 	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:478)
>> 	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseTypeInfos(TypeInfoUtils.java:305)
>> 	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils.getTypeInfosFromTypeString(TypeInfoUtils.java:754)
>> 	at org.apache.hadoop.hive.serde2.lazy.LazyUtils.extractColumnInfo(LazyUtils.java:372)
>> 	at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.initSerdeParams(LazySimpleSerDe.java:288)
>> 	at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.initialize(LazySimpleSerDe.java:187)
>> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genConversionSelectOperator(SemanticAnalyzer.java:5946)
>> 	... 26 more
>>
>>
>>
>> Thanks,
>> Binglin
>>
>
>


-- 
Thanks
Karan

Mime
View raw message