avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Pigott <mpigott.subscripti...@gmail.com>
Subject Re: How to get Specific classes to expose BigDecimal fields
Date Tue, 11 Nov 2014 04:06:35 GMT
Hi Fady,
    Properly handling BigDecimal types in Java is still an open question.
AVRO-1402 [1] added BigDecimal types to the Avro spec, but the Java support
is an open ticket under AVRO-1497 [2].  When I added BigDecimal support to
AVRO-457 (XML <-> Avro support), I added support for the Avro decimal
logical type using Java BigDecimals.  You can see the conversion code [3]
as well as the writer [4] and reader [5] code in my GitHub repository, or
download the patch in AVRO-457 [6] and look for BigDecimal in the
Utils.java, XmlDatumWriter.java, and XmlDatumReader.java files,
respectively.

Good luck!
Mike

[1] https://issues.apache.org/jira/browse/AVRO-1402
[2] https://issues.apache.org/jira/browse/AVRO-1497
[3]
https://github.com/mikepigott/xml-to-avro/blob/master/avro-to-xml/src/main/java/org/apache/avro/xml/Utils.java#L537
[4]
https://github.com/mikepigott/xml-to-avro/blob/master/avro-to-xml/src/main/java/org/apache/avro/xml/XmlDatumWriter.java#L1150
[5]
https://github.com/mikepigott/xml-to-avro/blob/master/avro-to-xml/src/main/java/org/apache/avro/xml/XmlDatumReader.java#L998
[6] https://issues.apache.org/jira/browse/AVRO-457

On Sat Nov 08 2014 at 4:11:32 AM Fady <fady@legsem.com> wrote:

> Hello,
>
> I am working on a project that aims at converting Mainframe data to Avro
> records (https://github.com/legsem/legstar.avro).
>
> Mainframe data often contains Decimal types. For these, I would like the
> corresponding Avro records to expose BigDecimal fields.
>
> Initially, I followed the recommendation here:
> http://avro.apache.org/docs/1.7.7/spec.html#Decimal. My schema contains
> for instance:
>
>      {
>        "name":"transactionAmount",
>        "type":{
>          "type":"bytes",
>          "logicalType":"decimal",
>          "precision":7,
>          "scale":2
>        }
>      }
>
> This works fine but the Avro Specific record produced by the
> SpecificCompiler exposes a ByteBuffer for that field.
>
>    @Deprecated public java.nio.ByteBuffer transactionAmount;
>
> Not what I want.
>
> I tried this alternative:
>
>      {
>        "name":"transactionAmount",
>        "type":{
>          "type":"string",
>          "java-class":"java.math.BigDecimal",
>          "logicalType":"decimal",
>          "precision":7,
>          "scale":2
>        }
>
> Now the SchemaCompiler produces the result I need:
>
>    @Deprecated public java.math.BigDecimal transactionAmount;
>
> There are 2 problems though:
>
> 1. It is less efficient to serialize/deserialize a BigDecimal from a
> string rather then the 2's complement.
>
> 2. The Specific Record obtained this way cannot be populated using a
> deep copy from a Generic Record.
>
> To clarify the second point:
>
> When I convert the mainframe data I do something like:
>
>          GenericRecord genericRecord = new GenericData.Record(schema);
>          ... populate genericRecord from Mainframe data ...
>          return (D) SpecificData.get().deepCopy(schema, genericRecord);
>
> This fails with :
>          java.lang.ClassCastException: java.lang.String cannot be cast
> to java.math.BigDecimal
>              at
> legstar.avro.test.specific.cusdat.Transaction.put(Transaction.java:47)
>              at
> org.apache.avro.generic.GenericData.setField(GenericData.java:573)
>              at
> org.apache.avro.generic.GenericData.setField(GenericData.java:590)
>              at
> org.apache.avro.generic.GenericData.deepCopy(GenericData.java:972)
>              at
> org.apache.avro.generic.GenericData.deepCopy(GenericData.java:926)
>              at
> org.apache.avro.generic.GenericData.deepCopy(GenericData.java:970)
>              at
> org.apache.avro.generic.GenericData.deepCopy(GenericData.java:970)
>
>
> This is because the code in the Specific record assumes the value
> received is already a BigDecimal
>
>      case 1: transactionAmount = (java.math.BigDecimal)value$; break;
>
> In other words, the java-class trick produces the right interface for
> Specific classes but the internal data types are not consistent with the
> GenericRecord derived from the same schema.
>
> So my question is: what would be a better approach for Specific classes
> to expose BigDecimal fields?
>
>

Mime
View raw message