hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anson Abraham <anson.abra...@gmail.com>
Subject Issues inserting Hive table in avro format...
Date Wed, 25 Apr 2012 14:39:21 GMT
In Hive, I'm having issues doing an insert overwrite table to a table that
is in avro format.

So my existing table (table1) is read from a hdfs directory where the files
are in avro format.

I created another table table2 in avro format (which is identical in
columns, etc...):

CREATE EXTERNAL TABLE IF NOT EXISTS table2
ROW FORMAT SERDE
'com.linkedin.haivvreo.AvroSerDe'
WITH SERDEPROPERTIES (
'schema.literal'='
{
"type" : "record",
"name" : "Record",
"namespace" : "GenericData",
"doc" : "table2/v1",
"fields" : [ {
"name" : "index_id",
"type" : "long"
},{
"name" : "id",
"type" : [ "null", "long" ],
"default" : null
} ]
}')
STORED AS INPUTFORMAT
'com.linkedin.haivvreo.AvroContainerInputFormat'
OUTPUTFORMAT
'com.linkedin.haivvreo.AvroContainerOutputFormat'
LOCATION 'hdfs://hivenode/table2/';


So when I did a

insert overwrite table table2
select * from table1


I get an error:

Ended Job = job_xxxxx_0123 with errors

FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask


Looking closely i see this:"

at org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:231)
	at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
	at org.apache.avro.io.ResolvingDecoder.readFieldOrder(ResolvingDecoder.java:127)
	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:169)
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144)
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:135)
	at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
	at org.apache.avro.file.DataFileStream.next(DataFileStream.java:220)
	at com.linkedin.haivvreo.AvroGenericRecordReader.next(AvroGenericRecordReader.java:116)
	at com.linkedin.haivvreo.AvroGenericRecordReader.next(AvroGenericRecordReader.java:41)
	at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:98)
	at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:42)
	at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
	at org.apache.hadoop.hive.shims.Hadoop20SShims$CombineFileRecordReader.next(Hadoop20SShims.java:208)
	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:208)
	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:193)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
	at org.apache.hadoop.mapred.Child.main(Child.java:264)

"

We're using
*haivvreo <https://github.com/jghoman/haivvreo>*
https://github.com/jghoman/haivvreo


Anyone have any ideas?

Mime
View raw message