avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1402) Support for DECIMAL primitive type
Date Mon, 31 Mar 2014 23:21:16 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13955862#comment-13955862

Xuefu Zhang commented on AVRO-1402:

The scale and precision specified in the column definition are maximums. My understanding
is that decimals with smaller scale and precision may be stored in such columns. For example
in Hive if I have a DECIMAL(5,3) column then 12.4 is stored as [124, 1] as the unscaled int-scale
pair (so precision 3, scale 1), and not as [12400, 3] So in effect there are per-record precision
and scale values, unless I'm missing something.

Sorry for being late on this. While it's possible that a column of type decimal might have
different scales from row to row, I'm not sure if there is any use case for that. At least,
majority of use cases is that all rows will have the same scale. So for majority of use cases,
storage efficiency will suffer a little for storing scale on per row basis. Most of time,
same application (such as Hive AvroSerde) will be used to write/read the data, so the way
data is stored can be controlled. Of course, storing scale per row is more generic. If that's
case, the need for maxPrecision/maxScale at schema is less meaningful, as consumer of the
decimal data will need and be able to figure out the precision/scale on a per-row basis.

If we do choose to storing scale per row, I'm wondering if byte instead of int can be used
as the type, saving some storage scale.

> Support for DECIMAL primitive type
> ----------------------------------
>                 Key: AVRO-1402
>                 URL: https://issues.apache.org/jira/browse/AVRO-1402
>             Project: Avro
>          Issue Type: New Feature
>    Affects Versions: 1.7.5
>            Reporter: Mariano Dominguez
>            Priority: Minor
>              Labels: Hive
>         Attachments: AVRO-1402.patch, AVRO-1402.patch, AVRO-1402.patch
> Currently, Avro does not seem to support a DECIMAL type or equivalent.
> http://avro.apache.org/docs/1.7.5/spec.html#schema_primitive
> Adding DECIMAL support would be particularly interesting when converting types from Avro
to Hive, since DECIMAL is already a supported data type in Hive (0.11.0).

This message was sent by Atlassian JIRA

View raw message