Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 44A26200BB1 for ; Thu, 3 Nov 2016 12:30:35 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 43338160AFF; Thu, 3 Nov 2016 11:30:35 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 420BF160AFE for ; Thu, 3 Nov 2016 12:30:34 +0100 (CET) Received: (qmail 42346 invoked by uid 500); 3 Nov 2016 11:30:33 -0000 Mailing-List: contact user-help@avro.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@avro.apache.org Delivered-To: mailing list user@avro.apache.org Received: (qmail 42336 invoked by uid 99); 3 Nov 2016 11:30:33 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Nov 2016 11:30:33 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 6282E1AA220 for ; Thu, 3 Nov 2016 11:30:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.999 X-Spam-Level: X-Spam-Status: No, score=-0.999 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RP_MATCHES_RCVD=-2.999, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id yYJPLg15LSWP for ; Thu, 3 Nov 2016 11:30:29 +0000 (UTC) Received: from mx9.sapient.com (mx9.sapient.com [12.130.39.183]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 916EE5FC3F for ; Thu, 3 Nov 2016 11:30:26 +0000 (UTC) Received: from unknown (HELO mx9.sapient.com) ([10.207.74.37]) by MX9int.sapient.com with ESMTP; 03 Nov 2016 07:30:18 -0400 Received: from DLUMAIL03.sapient.com ([fe80::2d6e:ea2d:ae54:adc0]) by DLUHUB03.sapient.com ([fe80::d99d:c0d5:2751:d7d0%27]) with mapi id 14.03.0210.002; Thu, 3 Nov 2016 17:00:17 +0530 From: Devendra Tomar To: "user@avro.apache.org" Subject: RE: Java / C++ interop and strings? Thread-Topic: Java / C++ interop and strings? Thread-Index: AdI1Ve5ikxbyB4N5ToSbtraR2sF1NgAbgBLA Date: Thu, 3 Nov 2016 11:30:17 +0000 Message-ID: <3A3737002FA2F645A8FBDE975DCE712429E0D7@DLUMAIL03.sapient.com> References: <869c103d7e1a44d18f3cf76ac7a700b3@AM3PR48MB0113.041d.mgd.msft.net> In-Reply-To: <869c103d7e1a44d18f3cf76ac7a700b3@AM3PR48MB0113.041d.mgd.msft.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.203.62.22] Content-Type: multipart/alternative; boundary="_000_3A3737002FA2F645A8FBDE975DCE712429E0D7DLUMAIL03sapientc_" MIME-Version: 1.0 archived-at: Thu, 03 Nov 2016 11:30:35 -0000 --_000_3A3737002FA2F645A8FBDE975DCE712429E0D7DLUMAIL03sapientc_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Tory, >> I've been testing using the 1.8.1 official release of both the Java and = C++ libraries. Any idea what's going on? I've been looking for some kind = of config options on the encoders/decoders for either language, didn't find >> much. Any specific additional info I should provide? In Java there is a class called CodecFactory that provides different compre= ssion codecs, you can find details here is the link: https://avro.apache.org/docs/1.8.1/api/java/org/apache/avro/file/CodecFacto= ry.html#method_summary Here is the use case of this particular class : https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kite-bundl= e/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/Conver= tJSONToAvro.java#L132 https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kite-bundl= e/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/Abstra= ctKiteConvertProcessor.java#L45 Hope it helps ! Regards Devendra Tomar From: MCKEAG Tory [mailto:tory.mckeag@alstom.com] Sent: Thursday, November 03, 2016 3:54 AM To: user@avro.apache.org Subject: Java / C++ interop and strings? Hi, I have some data I've serialized with Java and would like to deserializ= e in a C++ application, using GenericRecord in both languages. I've discov= ered that before each string field, the Java serialization is putting out a= n extra byte with the value of '2' (or the start text Unicode character? I= have no idea why this byte is there). After that byte, it contains the co= unt of characters in the string and of course the string data itself. The C++ decoder layer is choking on this extra byte. When I try to serial= ize the same data using C++, the extra byte isn't there and everything work= s. Also, when I deserialize the data written WITH the extra byte using Jav= a, it seems to work fine (I guess makes sense given the extra byte was adde= d by the Java serializer). I've been testing using the 1.8.1 official release of both the Java and C++= libraries. Any idea what's going on? I've been looking for some kind of = config options on the encoders/decoders for either language, didn't find mu= ch. Any specific additional info I should provide? ________________________________ CONFIDENTIALITY : This e-mail and any attachments are confidential and may = be privileged. If you are not a named recipient, please notify the sender i= mmediately and do not disclose the contents to another person, use it for a= ny purpose or store or copy the information in any medium. --_000_3A3737002FA2F645A8FBDE975DCE712429E0D7DLUMAIL03sapientc_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi Tory,

 

>> I’ve been testing using the 1.8.1 off= icial release of both the Java and C++ libraries.  Any idea wh= at’s going on?  I’ve been looking for some kind of config = options on the encoders/decoders for either language, didn’t find

>> much.  Any specific additional info I = should provide? 

 

In Java there is a class called CodecFactory that provides different compression = codecs, you can find details here is the link:

https://= avro.apache.org/docs/1.8.1/api/java/org/apache/avro/file/CodecFactory.html#= method_summary

 

Here is the use case of this particular cla= ss :

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kite-= bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/C= onvertJSONToAvro.java#L132

https://github.com/apache/nifi/blob/master/nifi-nar-b= undles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/= processors/kite/AbstractKiteConvertProcessor.java#L45=

 

Hope it helps !

 

Regards

Devendra Tomar

 

From: MCKEAG Tory [mailto:tory.mckeag@alstom.= com]
Sent: Thursday, November 03, 2016 3:54 AM
To: user@avro.apache.org
Subject: Java / C++ interop and strings?

 

Hi, I have some data I’ve serialized with Java= and would like to deserialize in a C++ application, using GenericR= ecord in both languages.  I’ve discovered that before each strin= g field, the Java serialization is putting out an extra byte with the value of ‘2’ (or the start text Unicode character?&nb= sp; I have no idea why this byte is there).  After that byte, it conta= ins the count of characters in the string and of course the string data its= elf. 

 

The C++ decoder layer is choking on this ext= ra byte.   When I try to serialize the same data using C++= ;, the extra byte isn’t there and everything works.  Also, when = I deserialize the data written WITH the extra byte using Java, it seems to work fine (I guess makes sense given the extra byte was added by the Ja= va serializer). 

 

I’ve been testing using the 1.8.1 official rel= ease of both the Java and C++ libraries.  Any idea what’= s going on?  I’ve been looking for some kind of config options o= n the encoders/decoders for either language, didn’t find much.  = Any specific additional info I should provide? 

 

CONFIDENTIALITY : This e-mail= and any attachments are confidential and may be privileged. If you are not= a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose= or store or copy the information in any medium.

--_000_3A3737002FA2F645A8FBDE975DCE712429E0D7DLUMAIL03sapientc_--