hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sagar Mehta <sagarme...@gmail.com>
Subject Bucketing issues with Hive table
Date Tue, 17 Dec 2013 00:01:13 GMT
Hi Guys,

We have an avro backed production table bucketed by a column and it works
fine with the old Hive = 0.7 on chd3u2.

Now we have moved to Hive 0.12 on HDP 2.0 and are getting bucketing issues
with this table.

Below is the create table statement

CREATE TABLE test_bucketing
  PARTITIONED BY (year INT, month INT, day INT, hour INT)
  CLUSTERED BY (akey) INTO 32 BUCKETS
  ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
  WITH SERDEPROPERTIES (
    'schema.literal'='{
  "type" : "record",
  "name" : "NgEventETLSimple",
  "namespace" : "com.ngmoco.ngpipes.etl.events",
  "fields" : [ {
    "name" : "akey",
    "type" : "string"
  } ]
}')
  STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
  OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'

and this is the error we are seeing

FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. Bucket columns akey is not part of
the table columns
([FieldSchema(name:error_error_error_error_error_error_error, type:string,
comment:from deserializer), FieldSchema(name:cannot_determine_schema,
type:string, comment:from deserializer), FieldSchema(name:check,
type:string, comment:from deserializer), FieldSchema(name:schema,
type:string, comment:from deserializer), FieldSchema(name:url, type:string,
comment:from deserializer), FieldSchema(name:and, type:string, comment:from
deserializer), FieldSchema(name:literal, type:string, comment:from
deserializer)]

As you can see the akey is mentioned as a part of the table definition,
however it still complains that akey is not part of the table definition.

I can confirm that independently the avro serde works fine [if we drop the
bucketing clause] and independently bucketing works fine on a test table
[without avro serde]. *However together bucketing for an avro backed table
is giving us issues on Hive 0.12*

Any idea what is going on? Any help/pointers would be useful.

Sagar
PS - The actual table in production has way more columns but I don't think
that is relevant for the issue at hand here.

Mime
View raw message