hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Jarcec Cecho (JIRA)" <>
Subject [jira] [Commented] (HIVE-7174) Do not accept string as scale and precision when reading Avro schema
Date Tue, 10 Jun 2014 14:01:34 GMT


Jarek Jarcec Cecho commented on HIVE-7174:

Thank you for bringing up this question [~leftylev]! I think that this particular JIRA is
just a bugfix that don't need explicit documentation. I've however noticed that the linked
wiki page do not contain any information about decimal in Avro that was introduced in HIVE-5823,
which is something that we should rectify. I'm not sure how to proceed though - should I create
a new doc JIRA for that or do you want to reuse the existing HIVE-5823?

> Do not accept string as scale and precision when reading Avro schema
> --------------------------------------------------------------------
>                 Key: HIVE-7174
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Jarek Jarcec Cecho
>            Assignee: Jarek Jarcec Cecho
>             Fix For: 0.14.0
>         Attachments: HIVE-7174.patch, dec.avro
> I've noticed that the current AvroSerde will happily accept schema that uses string instead
of integer for scale and precision, e.g. fragment {{"precision":"4","scale":"1"}} from following
> {code}
> CREATE TABLE `avro_dec1`(
>   `name` string COMMENT 'from deserializer',
>   `value` decimal(4,1) COMMENT 'from deserializer')
> COMMENT 'just drop the schema right into the HQL'
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   ''
>   ''
>   'numFiles'='1',
>   'avro.schema.literal'='{\"namespace\":\"com.howdy\",\"name\":\"some_schema\",\"type\":\"record\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"value\",\"type\":{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":\"4\",\"scale\":\"1\"}}]}'
> );
> {code}
> However the Decimal spec defined in AVRO-1402 requires only integer to be there and hence
is allowing only following fragment instead {{"precision":4,"scale":1}} (e.g. no double quotes
around numbers).
> As Hive can propagate this incorrect schema to new files and hence creating files with
invalid schema, I think that we should alter the behavior and insist on the correct schema.

This message was sent by Atlassian JIRA

View raw message