hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wayne Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-5820) Neither avro.schema.literal nor avro.schema.url specified, can't determine table schema
Date Fri, 04 Apr 2014 15:22:15 GMT

    [ https://issues.apache.org/jira/browse/HIVE-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13960042#comment-13960042
] 

Wayne Zhu commented on HIVE-5820:
---------------------------------

I have the same issue on CDH 4.3.0.
$ rpm -qa | grep hive
hive-hbase-0.10.0+121-1.cdh4.3.0.p0.16.el6.noarch
hive-metastore-0.10.0+121-1.cdh4.3.0.p0.16.el6.noarch
hive-jdbc-0.10.0+121-1.cdh4.3.0.p0.16.el6.noarch
hive-0.10.0+121-1.cdh4.3.0.p0.16.el6.noarch

Here is Hive steps:
hive> CREATE TABLE avro_price_internal
    > PARTITIONED BY (year int, month int, day int, hour int)
    > ROW FORMAT SERDE
    >   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
    > STORED AS INPUTFORMAT
    >   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
    > OUTPUTFORMAT
    >   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
    > TBLPROPERTIES (
    >   'avro.schema.url'='hdfs:///user/wzhu00/camus/camus-avsc/PriceSchema.avsc'
    > );
OK
Time taken: 2.762 seconds
hive> ALTER TABLE avro_price_internal ADD IF NOT EXISTS PARTITION (year=2014, month=03,
day=31, hour=19) LOCATION '/user/wzhu00/camus/dest/pricesingle/hourly/2014/03/31/19';
OK
Time taken: 0.595 seconds
hive> select * from avro_price_internal;
OK
Failed with exception java.io.IOException:org.apache.hadoop.hive.serde2.avro.BadSchemaException
Time taken: 0.463 seconds

> Neither avro.schema.literal nor avro.schema.url specified, can't determine table schema
> ---------------------------------------------------------------------------------------
>
>                 Key: HIVE-5820
>                 URL: https://issues.apache.org/jira/browse/HIVE-5820
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.10.0
>         Environment: CDH 4.3  Hive 0.10.0+121
>            Reporter: Sergey
>
> Hi, we've created a table:
> {code}
> create table tmp
> comment 'tmp'
> partitioned by (year string, month string, day string, fulldate string)
> row format serde 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> stored as
>     inputformat 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>     outputformat 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> location '/user/lol/tmp'
> tblproperties ('avro.schema.literal' =
>     '{"name": "tmp", "doc": "version 0.0.1", "type": "record", "fields": [
>         {"name": "a", "type": "int"},
>         {"name": "b", "type": "int"}
>     ]}'
> )
> {code}
> And we try to query it:
> {code}
> select * from tmp
> {code}
> and we get an exception
> {code}
> 13/11/14 17:12:15 WARN avro.AvroSerdeUtils: Encountered AvroSerdeException determining
schema. Returning signal schema to indicate problem
> org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Neither avro.schema.literal nor
avro.schema.url specified, can't determine table schema
> 	at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:66)
> 	at org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrReturnErrorSchema(AvroSerdeUtils.java:87)
> 	at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:59)
> 	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:249)
> 	at org.apache.hadoop.hive.ql.metadata.Partition.getDeserializer(Partition.java:251)
> 	at org.apache.hadoop.hive.ql.metadata.Partition.initialize(Partition.java:217)
> 	at org.apache.hadoop.hive.ql.metadata.Partition.<init>(Partition.java:107)
> 	at org.apache.hadoop.hive.ql.metadata.Hive.getPartitions(Hive.java:1573)
> 	at org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner.prune(PartitionPruner.java:190)
> 	at org.apache.hadoop.hive.ql.parse.ParseContext.getPrunedPartitions(ParseContext.java:561)
> 	at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.checkTree(SimpleFetchOptimizer.java:144)
> 	at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.optimize(SimpleFetchOptimizer.java:100)
> 	at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:74)
> 	at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:102)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8200)
> 	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:457)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
> 	at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.checkedCompile(BeeswaxServiceImpl.java:247)
> 	at com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.compile(BeeswaxServiceImpl.java:200)
> 	at com.cloudera.beeswax.BeeswaxServiceImpl$2.run(BeeswaxServiceImpl.java:830)
> 	at com.cloudera.beeswax.BeeswaxServiceImpl$2.run(BeeswaxServiceImpl.java:823)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> 	at com.cloudera.beeswax.BeeswaxServiceImpl.doWithState(BeeswaxServiceImpl.java:772)
> 	at com.cloudera.beeswax.BeeswaxServiceImpl.query(BeeswaxServiceImpl.java:822)
> 	at com.cloudera.beeswax.api.BeeswaxService$Processor$query.getResult(BeeswaxService.java:915)
> 	at com.cloudera.beeswax.api.BeeswaxService$Processor$query.getResult(BeeswaxService.java:899)
> 	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> 	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> 	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> 13/11/14 17:12:15 INFO parse.SemanticAnalyzer: Completed plan generation
> 13/11/14 17:12:15 INFO ql.Driver: Semantic Analysis Completed
> {code}
> here is describe:
> {code}
> 0	a	int	from deserializer
> 1	b	int	from deserializer
> 2	year	string	
> 3	month	string	
> 4	day	string	
> 5	fulldate	string	
> 6			
> 7	Detailed Table Information	Table(tableName:tmp, dbName:default, owner:devops, createTime:1384435112,
lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[], location:hdfs://nameservice1/user/fedyakov/tmp,
inputFormat:org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat,
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.avro.AvroSerDe,
parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[],
skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[FieldSchema(name:year,
type:string, comment:null), FieldSchema(name:month, type:string, comment:null), FieldSchema(name:day,
type:string, comment:null), FieldSchema(name:fulldate, type:string, comment:null)], parameters:{numPartitions=1,
numFiles=1, avro.schema.literal={"name": "tmp", "doc": "version 0.0.1", "type": "record",
"fields": [	
> 8	 {"name": "a", "type": "int"},		
> 9	 {"name": "b", "type": "int"}		
> 10	 ]}, transient_lastDdlTime=1384435137, numRows=0, totalSize=189, rawDataSize=0}, viewOriginalText:null,
viewExpandedText:null, tableType:MANAGED_TABLE)	
> {code}
> If we sepcify file instead of "embedded" avro schema, it works.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message