Return-Path: X-Original-To: apmail-avro-user-archive@www.apache.org Delivered-To: apmail-avro-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2D215179DF for ; Wed, 12 Nov 2014 03:13:34 +0000 (UTC) Received: (qmail 76826 invoked by uid 500); 12 Nov 2014 03:13:33 -0000 Delivered-To: apmail-avro-user-archive@avro.apache.org Received: (qmail 76746 invoked by uid 500); 12 Nov 2014 03:13:33 -0000 Mailing-List: contact user-help@avro.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@avro.apache.org Delivered-To: mailing list user@avro.apache.org Received: (qmail 76736 invoked by uid 99); 12 Nov 2014 03:13:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Nov 2014 03:13:33 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mpigott.subscriptions@gmail.com designates 209.85.213.194 as permitted sender) Received: from [209.85.213.194] (HELO mail-ig0-f194.google.com) (209.85.213.194) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Nov 2014 03:13:07 +0000 Received: by mail-ig0-f194.google.com with SMTP id r2so477993igi.1 for ; Tue, 11 Nov 2014 19:11:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:references:from:date:message-id:subject:to :content-type; bh=6ELcPjNwILW/7RyIFSHGS3vCRg6IhdEYzRdtH51fU3U=; b=JoJA5qlEeyT6cnIXsXWT/9JXpGcGeF0dWurNKFutFoUpPnVz3HwrsCsFFT4MuEgsJp wI5gmsqeWDMkxfIGylW8htTyiUV6t3YpOb09tyohHk5FfIlXII3pBfKZ3F+G6r2CjswR Ol7ZYj0cDGeG3FXLocN12gNn567jNHlOyd2MBiskqdpSmzp0281RK1ocNv/LMcIuypYl SJqbg/qq9dO7/XdBc5EZ45xbruI1HVKCIwvQHDVoIiIp4AZNGPrWJZynBhyfrqwVaLo3 I4n6zzn94UeOpx6p079vKuUotCRL19wykrE2ZBFeAY9vMz95sq5dzw6dizne6hk/xXcz jkug== X-Received: by 10.43.89.8 with SMTP id bc8mr45388150icc.35.1415761895845; Tue, 11 Nov 2014 19:11:35 -0800 (PST) MIME-Version: 1.0 References: <545DDDD4.8070201@legsem.com> <5461CC46.2050001@legsem.com> From: Michael Pigott Date: Wed, 12 Nov 2014 03:11:35 +0000 Message-ID: Subject: Re: How to get Specific classes to expose BigDecimal fields To: user@avro.apache.org Content-Type: multipart/alternative; boundary=bcaec5186658e37a5b0507a0c035 X-Virus-Checked: Checked by ClamAV on apache.org --bcaec5186658e37a5b0507a0c035 Content-Type: text/plain; charset=UTF-8 You're welcome! I'm glad I was able to help. If you find a better long-term solution, feel free to offer it to AVRO-1497! -Mike On Tue Nov 11 2014 at 3:44:26 AM Fady wrote: > > Thank you Mike for taking the time to reply to this, > > I looked at your code and applied the AVRO-457 patch you did. Indeed you > fixed a very similar problem. In your case XMLSchema delivers and expects > BigDecimals so you mapped that to ByteBuffer as specified in > http://avro.apache.org/docs/1.7.7/spec.html#Decimal. > > As for the SpecificCompiler, I ended up creating s custom compiler: > > /** > * Temporary workaround for the lack of support for BigDecimal in > Avro Specific Compiler. > *

> * The record.vm template is customized to expose BigDecimal getters > and setters. > * > */ > public class CustomSpecificCompiler extends SpecificCompiler { > > private static final String TEMPLATES_PATH = > "/com/legstar/avro/generator/specific/templates/java/classic/"; > > public CustomSpecificCompiler(Schema schema) { > super(schema); > setTemplateDir(TEMPLATES_PATH); > } > > /** > * In the case of BigDecimals there is an internal java type > (ByteBuffer) > * and an external java type for getters/setters. > * > * @param schema the field schema > * @return the field java type > */ > public String externalJavaType(Schema schema) { > return isBigDecimal(schema) ? "java.math.BigDecimal" : super > .javaType(schema); > } > > /** Tests whether a field is to be externalized as a BigDecimal > */ > public static boolean isBigDecimal(Schema schema) { > if (Type.BYTES == schema.getType()) { > JsonNode logicalTypeNode = > schema.getJsonProp("logicalType"); > if (logicalTypeNode != null > && "decimal".equals(logicalTypeNode.asText())) { > return true; > } > } > return false; > } > > } > > And then changed the record.vm velocity template like this: > > 72c72 > < public ${this.mangle($schema.getName())}(#foreach($field in > $schema.getFields())${this.externalJavaType($field.schema())} > ${this.mangle($field.name())}#if($velocityCount < > $schema.getFields().size()), #end#end) { > --- > > public ${this.mangle($schema.getName())}(#foreach($field in > $schema.getFields())${this.javaType($field.schema())} ${this.mangle($ > field.name())}#if($velocityCount < $schema.getFields().size()), #end#end) > { > 74c74 > < ${this.generateSetMethod($schema, $field)}(${this.mangle($ > field.name())}); > --- > > this.${this.mangle($field.name())} = ${this.mangle($field.name > ())}; > 110,113c110 > < public ${this.externalJavaType($field.schema())} > ${this.generateGetMethod($schema, $field)}() { > < #if ($this.isBigDecimal($field.schema())) > < return new java.math.BigDecimal(new > java.math.BigInteger(${this.mangle($field.name())}.array()), > $field.schema().getJsonProp("scale")); > < #else > --- > > public ${this.javaType($field.schema())} > ${this.generateGetMethod($schema, $field)}() { > 115d111 > < #end > 124,127c120 > < public void ${this.generateSetMethod($schema, > $field)}(${this.externalJavaType($field.schema())} value) { > < #if ($this.isBigDecimal($field.schema())) > < this.${this.mangle($field.name(), $schema.isError())} = > java.nio.ByteBuffer.wrap(value.unscaledValue().toByteArray()); > < #else > --- > > public void ${this.generateSetMethod($schema, > $field)}(${this.javaType($field.schema())} value) { > 129d121 > < #end > > This fixes the issue for me but is not a good long term solution. > Particularly the builder part of the generated Specific class is still > exposing ByteBuffer instead of BigDecimal which is inconsistent. > > More generally, it seems to me a better solution would be that the > "java-class" trick be extended so that more complex conversions can occur > between the avro type and the java type exposed by Specific classes. Right > now, the java type must be castable from the avro type which is limiting. > > Anyway, thanks again for your great insight. > > > Fady > > > > > > On 11/11/2014 05:06, Michael Pigott wrote: > > Hi Fady, > Properly handling BigDecimal types in Java is still an open question. > AVRO-1402 [1] added BigDecimal types to the Avro spec, but the Java support > is an open ticket under AVRO-1497 [2]. When I added BigDecimal support to > AVRO-457 (XML <-> Avro support), I added support for the Avro decimal > logical type using Java BigDecimals. You can see the conversion code [3] > as well as the writer [4] and reader [5] code in my GitHub repository, or > download the patch in AVRO-457 [6] and look for BigDecimal in the > Utils.java, XmlDatumWriter.java, and XmlDatumReader.java files, > respectively. > > Good luck! > Mike > > [1] https://issues.apache.org/jira/browse/AVRO-1402 > [2] https://issues.apache.org/jira/browse/AVRO-1497 > [3] > https://github.com/mikepigott/xml-to-avro/blob/master/avro-to-xml/src/main/java/org/apache/avro/xml/Utils.java#L537 > [4] > https://github.com/mikepigott/xml-to-avro/blob/master/avro-to-xml/src/main/java/org/apache/avro/xml/XmlDatumWriter.java#L1150 > [5] > https://github.com/mikepigott/xml-to-avro/blob/master/avro-to-xml/src/main/java/org/apache/avro/xml/XmlDatumReader.java#L998 > [6] https://issues.apache.org/jira/browse/AVRO-457 > > On Sat Nov 08 2014 at 4:11:32 AM Fady wrote: > >> Hello, >> >> I am working on a project that aims at converting Mainframe data to Avro >> records (https://github.com/legsem/legstar.avro). >> >> Mainframe data often contains Decimal types. For these, I would like the >> corresponding Avro records to expose BigDecimal fields. >> >> Initially, I followed the recommendation here: >> http://avro.apache.org/docs/1.7.7/spec.html#Decimal. My schema contains >> for instance: >> >> { >> "name":"transactionAmount", >> "type":{ >> "type":"bytes", >> "logicalType":"decimal", >> "precision":7, >> "scale":2 >> } >> } >> >> This works fine but the Avro Specific record produced by the >> SpecificCompiler exposes a ByteBuffer for that field. >> >> @Deprecated public java.nio.ByteBuffer transactionAmount; >> >> Not what I want. >> >> I tried this alternative: >> >> { >> "name":"transactionAmount", >> "type":{ >> "type":"string", >> "java-class":"java.math.BigDecimal", >> "logicalType":"decimal", >> "precision":7, >> "scale":2 >> } >> >> Now the SchemaCompiler produces the result I need: >> >> @Deprecated public java.math.BigDecimal transactionAmount; >> >> There are 2 problems though: >> >> 1. It is less efficient to serialize/deserialize a BigDecimal from a >> string rather then the 2's complement. >> >> 2. The Specific Record obtained this way cannot be populated using a >> deep copy from a Generic Record. >> >> To clarify the second point: >> >> When I convert the mainframe data I do something like: >> >> GenericRecord genericRecord = new GenericData.Record(schema); >> ... populate genericRecord from Mainframe data ... >> return (D) SpecificData.get().deepCopy(schema, genericRecord); >> >> This fails with : >> java.lang.ClassCastException: java.lang.String cannot be cast >> to java.math.BigDecimal >> at >> legstar.avro.test.specific.cusdat.Transaction.put(Transaction.java:47) >> at >> org.apache.avro.generic.GenericData.setField(GenericData.java:573) >> at >> org.apache.avro.generic.GenericData.setField(GenericData.java:590) >> at >> org.apache.avro.generic.GenericData.deepCopy(GenericData.java:972) >> at >> org.apache.avro.generic.GenericData.deepCopy(GenericData.java:926) >> at >> org.apache.avro.generic.GenericData.deepCopy(GenericData.java:970) >> at >> org.apache.avro.generic.GenericData.deepCopy(GenericData.java:970) >> >> >> This is because the code in the Specific record assumes the value >> received is already a BigDecimal >> >> case 1: transactionAmount = (java.math.BigDecimal)value$; break; >> >> In other words, the java-class trick produces the right interface for >> Specific classes but the internal data types are not consistent with the >> GenericRecord derived from the same schema. >> >> So my question is: what would be a better approach for Specific classes >> to expose BigDecimal fields? >> >> > --bcaec5186658e37a5b0507a0c035 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable You're welcome!=C2=A0 I'm glad I was able to help.=C2=A0 If you fin= d a better long-term solution, feel free to offer it to AVRO-1497! =C2=A0-M= ike

On Tue Nov 11 2014 at 3:44:26 AM Fady= <fady@legsem.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex"> =20 =20 =20

Thank you Mike for taking the time to reply to this,

I looked at your code and applied the AVRO-457 patch you did. Indeed you fixed a very similar problem. In your case XMLSchema delivers and expects BigDecimals so you mapped that to ByteBuffer as specified in http://avro.apache.org/docs/1.7.7/spec.html#Decimal.

As for the SpecificCompiler, I ended up creating s custom compiler:

=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 /**
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * Temporary workaround for the= lack of support for BigDecimal in Avro Specific Compiler.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * <p/>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * The record.vm template is cu= stomized to expose BigDecimal getters and setters.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 */
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 public class CustomSpecificCompiler = extends SpecificCompiler {
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 private stat= ic final String TEMPLATES_PATH =3D "/com/legstar/avro/generator/specific/templates/java/classic/&= quot;;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 public Custo= mSpecificCompiler(Schema schema) {
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 super(schema);
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 setTemplateDir(TEMPLATES_PATH);
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /**
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * In t= he case of BigDecimals there is an internal java type (ByteBuffer)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * and = an external java type for getters/setters.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * @par= am schema the field schema
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * @ret= urn the field java type
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 */
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 public Strin= g externalJavaType(Schema schema) {
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 return isBigDecimal(schema) ? "java.math.BigDecimal" : super
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .javaType(sche= ma);
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /** Tests wh= ether a field is to be externalized as a BigDecimal */
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 public stati= c boolean isBigDecimal(Schema schema) {
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 if (Type.BYTES =3D=3D schema.getType()) {
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 JsonNode logicalTypeNode =3D schema.getJsonProp("logicalType");
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (logicalTypeNode !=3D null
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 && "decimal".equals(logicalTypeNode.asText())) {
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return true; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 }
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 return false;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }

And then changed the record.vm velocity template like this:

=C2=A0=C2=A0=C2=A0 72c= 72
=C2=A0=C2=A0=C2=A0 <=C2=A0=C2=A0 public ${this.mangle($schema.getName())}(#foreach($field in $schema.getFields())${this.externalJavaType($field.schema())} ${this.mangle($fiel= d.name())}#if($velocityCount < $schema.getFields().size()), #end#end) {
=C2=A0=C2=A0=C2=A0 ---
=C2=A0=C2=A0=C2=A0 >=C2=A0=C2=A0 public ${this.mangle($schema.getName())}(#foreach($field in $schema.getFields())${this.javaType($field.schema())} ${this.mangle($fiel= d.name())}#if($velocityCount < $schema.getFields().size()), #end#end) {
=C2=A0=C2=A0=C2=A0 74c74
=C2=A0=C2=A0=C2=A0 <=C2=A0=C2=A0=C2=A0=C2=A0 ${this.generateSetM= ethod($schema, $field)}(${this.mangle($field.name())});
=C2=A0=C2=A0=C2=A0 ---
=C2=A0=C2=A0=C2=A0 >=C2=A0=C2=A0=C2=A0=C2=A0 this.${this.mangle(= $field.name())} =3D ${this.mangle($fiel= d.name())};
=C2=A0=C2=A0=C2=A0 110,113c110
=C2=A0=C2=A0=C2=A0 <=C2=A0=C2=A0 public ${this.externalJavaType(= $field.schema())} ${this.generateGetMethod($schema, $field)}() {
=C2=A0=C2=A0=C2=A0 < #if ($this.isBigDecimal($field.schema())) =C2=A0=C2=A0=C2=A0 <=C2=A0=C2=A0=C2=A0=C2=A0 return new java.mat= h.BigDecimal(new java.math.BigInteger(${this.mangle($field.name())}.array()), $field.schema().getJsonProp("scale"));
=C2=A0=C2=A0=C2=A0 < #else
=C2=A0=C2=A0=C2=A0 ---
=C2=A0=C2=A0=C2=A0 >=C2=A0=C2=A0 public ${this.javaType($field.s= chema())} ${this.generateGetMethod($schema, $field)}() {
=C2=A0=C2=A0=C2=A0 115d111
=C2=A0=C2=A0=C2=A0 < #end
=C2=A0=C2=A0=C2=A0 124,127c120
=C2=A0=C2=A0=C2=A0 <=C2=A0=C2=A0 public void ${this.generateSetM= ethod($schema, $field)}(${this.externalJavaType($field.schema())} value) {
=C2=A0=C2=A0=C2=A0 < #if ($this.isBigDecimal($field.schema())) =C2=A0=C2=A0=C2=A0 <=C2=A0=C2=A0=C2=A0=C2=A0 this.${this.mangle(= $field.name(), $schema.isError())} =3D java.nio.ByteBuffer.wrap(value.unscaledValue().toByteArray());
=C2=A0=C2=A0=C2=A0 < #else
=C2=A0=C2=A0=C2=A0 ---
=C2=A0=C2=A0=C2=A0 >=C2=A0=C2=A0 public void ${this.generateSetM= ethod($schema, $field)}(${this.javaType($field.schema())} value) {
=C2=A0=C2=A0=C2=A0 129d121
=C2=A0=C2=A0=C2=A0 < #end

This fixes the issue for me but is not a good long term solution. Particularly the builder part of the generated Specific class is still exposing ByteBuffer instead of BigDecimal which is inconsistent.

More generally, it seems to me a better solution would be that the "java-class" trick be extended so that more complex convers= ions can occur between the avro type and the java type exposed by Specific classes. Right now, the java type must be castable from the avro type which is limiting.

Anyway, thanks again for your great insight.


Fady





On 11/11/2014 05:06, Michael Pigott wrote:
Hi Fady,
=C2=A0 =C2=A0 Properly handling BigDecimal types in Java is still an = open question.=C2=A0 AVRO-1402 [1] added BigDecimal types to the Avro spec= , but the Java support is an open ticket under AVRO-1497 [2].=C2=A0 Whe= n I added BigDecimal support to AVRO-457 (XML <-> Avro support), I added support for the Avro decimal logical type using Java BigDecimals.=C2=A0 You can see the conversion code [3] as well a= s the writer [4] and reader [5] code in my GitHub repository, or download the patch in AVRO-457 [6] and look for BigDecimal in the Utils.java, XmlDatumWriter.java, and XmlDatumReader.java files, respectively.

Good luck!
Mike


On Sat Nov 08 2014 at 4:11:32 AM Fady <fady@legse= m.com> wrote:
Hello,

I am working on a project that aims at converting Mainframe data to Avro
records (https://github.com/legsem/legstar.avro).

Mainframe data often contains Decimal types. For these, I would like the
corresponding Avro records to expose BigDecimal fields.

Initially, I followed the recommendation here:
http://avro.apache.org/docs/1.7.7/spec.html#Decimal. My schema contains
for instance:

=C2=A0 =C2=A0 =C2=A0{
=C2=A0 =C2=A0 =C2=A0 =C2=A0"name":"transactionAmou= nt",
=C2=A0 =C2=A0 =C2=A0 =C2=A0"type":{
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"type":"bytes&qu= ot;,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"logicalType":"d= ecimal",
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"precision":7,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"scale":2
=C2=A0 =C2=A0 =C2=A0 =C2=A0}
=C2=A0 =C2=A0 =C2=A0}

This works fine but the Avro Specific record produced by the
SpecificCompiler exposes a ByteBuffer for that field.

=C2=A0 =C2=A0@Deprecated public java.nio.ByteBuffer transactionAm= ount;

Not what I want.

I tried this alternative:

=C2=A0 =C2=A0 =C2=A0{
=C2=A0 =C2=A0 =C2=A0 =C2=A0"name":"transactionAmou= nt",
=C2=A0 =C2=A0 =C2=A0 =C2=A0"type":{
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"type":"string&q= uot;,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"java-class":"ja= va.math.BigDecimal",
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"logicalType":"d= ecimal",
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"precision":7,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"scale":2
=C2=A0 =C2=A0 =C2=A0 =C2=A0}

Now the SchemaCompiler produces the result I need:

=C2=A0 =C2=A0@Deprecated public java.math.BigDecimal transactionA= mount;

There are 2 problems though:

1. It is less efficient to serialize/deserialize a BigDecimal from a
string rather then the 2's complement.

2. The Specific Record obtained this way cannot be populated using a
deep copy from a Generic Record.

To clarify the second point:

When I convert the mainframe data I do something like:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0GenericRecord genericRecord =3D= new GenericData.Record(schema);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0... populate genericRecord from= Mainframe data ...
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return (D) SpecificData.get().d= eepCopy(schema, genericRecord);

This fails with :
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0java.lang.ClassCastException: j= ava.lang.String cannot be cast
to java.math.BigDecimal
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
legstar.avro.test.specific.cusdat.Transaction.put(Transaction.jav= a:47)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.avro.generic.GenericData.setField(GenericData.java:573= )
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.avro.generic.GenericData.setField(GenericData.java:590= )
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.avro.generic.GenericData.deepCopy(GenericData.java:972= )
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.avro.generic.GenericData.deepCopy(GenericData.java:926= )
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.avro.generic.GenericData.deepCopy(GenericData.java:970= )
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
org.apache.avro.generic.GenericData.deepCopy(GenericData.java:970= )


This is because the code in the Specific record assumes the value
received is already a BigDecimal

=C2=A0 =C2=A0 =C2=A0case 1: transactionAmount =3D (java.math.BigD= ecimal)value$; break;

In other words, the java-class trick produces the right interface for
Specific classes but the internal data types are not consistent with the
GenericRecord derived from the same schema.

So my question is: what would be a better approach for Specific classes
to expose BigDecimal fields?


--bcaec5186658e37a5b0507a0c035--