avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fady <f...@legsem.com>
Subject How to get Specific classes to expose BigDecimal fields
Date Sat, 08 Nov 2014 09:09:40 GMT
Hello,

I am working on a project that aims at converting Mainframe data to Avro 
records (https://github.com/legsem/legstar.avro).

Mainframe data often contains Decimal types. For these, I would like the 
corresponding Avro records to expose BigDecimal fields.

Initially, I followed the recommendation here: 
http://avro.apache.org/docs/1.7.7/spec.html#Decimal. My schema contains 
for instance:

     {
       "name":"transactionAmount",
       "type":{
         "type":"bytes",
         "logicalType":"decimal",
         "precision":7,
         "scale":2
       }
     }

This works fine but the Avro Specific record produced by the 
SpecificCompiler exposes a ByteBuffer for that field.

   @Deprecated public java.nio.ByteBuffer transactionAmount;

Not what I want.

I tried this alternative:

     {
       "name":"transactionAmount",
       "type":{
         "type":"string",
         "java-class":"java.math.BigDecimal",
         "logicalType":"decimal",
         "precision":7,
         "scale":2
       }

Now the SchemaCompiler produces the result I need:

   @Deprecated public java.math.BigDecimal transactionAmount;

There are 2 problems though:

1. It is less efficient to serialize/deserialize a BigDecimal from a 
string rather then the 2's complement.

2. The Specific Record obtained this way cannot be populated using a 
deep copy from a Generic Record.

To clarify the second point:

When I convert the mainframe data I do something like:

         GenericRecord genericRecord = new GenericData.Record(schema);
         ... populate genericRecord from Mainframe data ...
         return (D) SpecificData.get().deepCopy(schema, genericRecord);

This fails with :
         java.lang.ClassCastException: java.lang.String cannot be cast 
to java.math.BigDecimal
             at 
legstar.avro.test.specific.cusdat.Transaction.put(Transaction.java:47)
             at 
org.apache.avro.generic.GenericData.setField(GenericData.java:573)
             at 
org.apache.avro.generic.GenericData.setField(GenericData.java:590)
             at 
org.apache.avro.generic.GenericData.deepCopy(GenericData.java:972)
             at 
org.apache.avro.generic.GenericData.deepCopy(GenericData.java:926)
             at 
org.apache.avro.generic.GenericData.deepCopy(GenericData.java:970)
             at 
org.apache.avro.generic.GenericData.deepCopy(GenericData.java:970)


This is because the code in the Specific record assumes the value 
received is already a BigDecimal

     case 1: transactionAmount = (java.math.BigDecimal)value$; break;

In other words, the java-class trick produces the right interface for 
Specific classes but the internal data types are not consistent with the 
GenericRecord derived from the same schema.

So my question is: what would be a better approach for Specific classes 
to expose BigDecimal fields?


Mime
View raw message