avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From MCKEAG Tory <tory.mck...@alstom.com>
Subject RE: Java / C++ interop and strings?
Date Thu, 03 Nov 2016 16:38:24 GMT
Thanks Devendra, that's helpful.  However, I'm not using any compression, and I still have
these extra bytes when serializing from Java.  Any clues?

From: Devendra Tomar [mailto:dtomar2@sapient.com]
Sent: Thursday, November 03, 2016 4:30 AM
To: user@avro.apache.org
Subject: RE: Java / C++ interop and strings?

Hi Tory,

>> I've been testing using the 1.8.1 official release of both the Java and C++ libraries.
 Any idea what's going on?  I've been looking for some kind of config options on the encoders/decoders
for either language, didn't find
>> much.  Any specific additional info I should provide?

In Java there is a class called CodecFactory that provides different compression codecs, you
can find details here is the link:
https://avro.apache.org/docs/1.8.1/api/java/org/apache/avro/file/CodecFactory.html#method_summary<https://urldefense.proofpoint.com/v2/url?u=https-3A__avro.apache.org_docs_1.8.1_api_java_org_apache_avro_file_CodecFactory.html-23method-5Fsummary&d=DQMFAg&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=KrzgCoTCVkLR0UvTgzkp8wgH6Y5iH32djW6rC-0fU6E&m=Ft9c8IFVvoZ_z3Wi4-o7NUBZ1IGODnpBQMdFKNDdxAw&s=vB_DyvjFjc5o5OKVNqEz-RlmJ0oOu1TZmQePfgjHE4w&e=>

Here is the use case of this particular class :
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/ConvertJSONToAvro.java#L132<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Dnar-2Dbundles_nifi-2Dkite-2Dbundle_nifi-2Dkite-2Dprocessors_src_main_java_org_apache_nifi_processors_kite_ConvertJSONToAvro.java-23L132&d=DQMFAg&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=KrzgCoTCVkLR0UvTgzkp8wgH6Y5iH32djW6rC-0fU6E&m=Ft9c8IFVvoZ_z3Wi4-o7NUBZ1IGODnpBQMdFKNDdxAw&s=CjE53xnUl2R38C_ZQA0Xpw9RsX3Xo-S963lXt-6VcHQ&e=>
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/AbstractKiteConvertProcessor.java#L45<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_nifi_blob_master_nifi-2Dnar-2Dbundles_nifi-2Dkite-2Dbundle_nifi-2Dkite-2Dprocessors_src_main_java_org_apache_nifi_processors_kite_AbstractKiteConvertProcessor.java-23L45&d=DQMFAg&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=KrzgCoTCVkLR0UvTgzkp8wgH6Y5iH32djW6rC-0fU6E&m=Ft9c8IFVvoZ_z3Wi4-o7NUBZ1IGODnpBQMdFKNDdxAw&s=xOg8URZV_CyqszM0C55gUi0flKGDXksZcIwYdrw1Vi8&e=>

Hope it helps !

Regards
Devendra Tomar

From: MCKEAG Tory [mailto:tory.mckeag@alstom.com]
Sent: Thursday, November 03, 2016 3:54 AM
To: user@avro.apache.org<mailto:user@avro.apache.org>
Subject: Java / C++ interop and strings?

Hi, I have some data I've serialized with Java and would like to deserialize in a C++ application,
using GenericRecord in both languages.  I've discovered that before each string field, the
Java serialization is putting out an extra byte with the value of '2' (or the start text Unicode
character?  I have no idea why this byte is there).  After that byte, it contains the count
of characters in the string and of course the string data itself.

The C++ decoder layer is choking on this extra byte.   When I try to serialize the same data
using C++, the extra byte isn't there and everything works.  Also, when I deserialize the
data written WITH the extra byte using Java, it seems to work fine (I guess makes sense given
the extra byte was added by the Java serializer).

I've been testing using the 1.8.1 official release of both the Java and C++ libraries.  Any
idea what's going on?  I've been looking for some kind of config options on the encoders/decoders
for either language, didn't find much.  Any specific additional info I should provide?

________________________________
CONFIDENTIALITY : This e-mail and any attachments are confidential and may be privileged.
If you are not a named recipient, please notify the sender immediately and do not disclose
the contents to another person, use it for any purpose or store or copy the information in
any medium.

________________________________
CONFIDENTIALITY : This e-mail and any attachments are confidential and may be privileged.
If you are not a named recipient, please notify the sender immediately and do not disclose
the contents to another person, use it for any purpose or store or copy the information in
any medium.

Mime
View raw message