Return-Path: X-Original-To: apmail-avro-user-archive@www.apache.org Delivered-To: apmail-avro-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0634A10133 for ; Mon, 17 Jun 2013 05:46:56 +0000 (UTC) Received: (qmail 57044 invoked by uid 500); 17 Jun 2013 05:46:52 -0000 Delivered-To: apmail-avro-user-archive@avro.apache.org Received: (qmail 56970 invoked by uid 500); 17 Jun 2013 05:46:50 -0000 Mailing-List: contact user-help@avro.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@avro.apache.org Delivered-To: mailing list user@avro.apache.org Received: (qmail 56962 invoked by uid 99); 17 Jun 2013 05:46:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jun 2013 05:46:48 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: unknown (athena.apache.org: error in processing during lookup of martin@rapportive.com) Received: from [209.85.212.50] (HELO mail-vb0-f50.google.com) (209.85.212.50) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jun 2013 05:46:43 +0000 Received: by mail-vb0-f50.google.com with SMTP id w16so1662755vbb.9 for ; Sun, 16 Jun 2013 22:46:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:in-reply-to:references:from:date :message-id:subject:to:content-type:x-gm-message-state; bh=b4O5bJbBL1Ei6hE1bEsC1atcrAPH6Y6Tk9kQ7OVR4lo=; b=Eigf2cY3vAsP9Ss6XRWr7iarAM90GaI6pN4QWgLSHqXD2iXJSmwPEZn7LH+mAUhiAK NnsMc398OJFQhQOgIlfmNWUssExDCOVcKCz/SlEwhqpgrz3fN1o/rKjHgl8iOACd4Ftc VNsLp1sCTxyDCzkAK2zgXyIBaS77Z2UTFMOxkEAUbClLLNt8CcPgeRa+rHDhtPaRMp0A cYotnes1Bxp1NupMkWKKcJZ6o4ZoXR4GsvmsdZWsOg/BRgWdvtDyBjQ4qT3WUH6D1CvD zhWXP5Cc83OFvYEs/DYy5k8NP3GVwfI6N0u3xcGvMmrUbfVfO20mZvwh94tmw5TKDKrv qJNg== X-Received: by 10.220.59.69 with SMTP id k5mr3283172vch.34.1371447981936; Sun, 16 Jun 2013 22:46:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.52.232.103 with HTTP; Sun, 16 Jun 2013 22:46:00 -0700 (PDT) X-Originating-IP: [50.0.150.94] In-Reply-To: References: From: Martin Kleppmann Date: Sun, 16 Jun 2013 22:46:00 -0700 Message-ID: Subject: Re: Sending Apache avro serialized data to thrift server To: user@avro.apache.org Content-Type: multipart/alternative; boundary=001a11c2012ccc069604df531d39 X-Gm-Message-State: ALoCoQmLJ7Jlt6iVYj2YJdaQvIf2ROUBD9Ou/s6J73HlXt/lfg4GwX2LYSxQjOHyyfsXR1Q1sdKt X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2012ccc069604df531d39 Content-Type: text/plain; charset=ISO-8859-1 It's hard to tell just by looking at the code -- I suggest you inspect the contents of the array you're decoding in each case, and see if there is any difference. Purely guessing, the ByteBuffer that Thrift gives you may have other data before and after the Avro record, and calling `bf.array()` will give you junk in that case. You may need something more like this (rough outline, please read the API docs): byte[] copy = new byte[bf.limit() - bf.position()]; bf.read(copy); DecoderFactory.get().binaryDecoder(copy, null); On 16 June 2013 20:15, Avinash Dongre wrote: > > Hi, >> I am trying to send avro serialized data to my thrift server. But when I >> try to de-serialized I get exception >> org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: >> -51 >> >> If I do the same operation in the same vm process everything works fine. >> >> Here is my small code. >> >> public static GenericRecord createContentNestedObject() throws Exception { >> GenericRecord image1 = new GenericData.Record(IMAGE_SCHEMA); >> image1.put("uri", new Utf8("http://javaone.com/keynote_large.jpg")); >> image1.put("width", 0); >> image1.put("height", 0); >> image1.put("size", 2); >> image1.put("title", new Utf8("Javaone Keynote")); >> return image1; >> } >> >> // Helper Method to serialize the object from avro to bytebuffer >> public static ByteBuffer serialize(GenericRecord content, Schema schema) >> throws Exception { >> >> ByteArrayOutputStream out = new ByteArrayOutputStream(); >> DatumWriter writer = new >> GenericDatumWriter( >> schema); >> // create Binary Encoder >> EncoderFactory ef = new EncoderFactory(); >> BinaryEncoder be = ef.binaryEncoder(out, null); >> writer.write(content, be); >> be.flush(); >> out.close(); >> >> return ByteBuffer.wrap(out.toByteArray()); >> } >> >> public static void main(String[] args) throws Exception { >> try { >> GenericRecord rd = createContentNestedObject(); >> bf = serialize(rd, IMAGE_SCHEMA); >> >> } catch (Exception e) { >> // TODO Auto-generated catch block >> e.printStackTrace(); >> } >> >> // Deserialization. >> DatumReader reader = new >> GenericDatumReader(IMAGE_SCHEMA_1); >> ByteArrayOutputStream out = new ByteArrayOutputStream(); >> try { >> out.write(bf.array()); >> } catch (IOException e1) { >> // TODO Auto-generated catch block >> e1.printStackTrace(); >> } >> BinaryDecoder decoder = >> DecoderFactory.get().binaryDecoder(out.toByteArray(), null); >> try { >> GenericRecord result = reader.read(null, decoder); >> System.out.println("RESULT : " + result.toString()); >> } catch (IOException e) { >> // TODO Auto-generated catch block >> e.printStackTrace(); >> } >> >> This code works. When I send the bf to my thrift api , I am getting the >> error, >> my thrift api is simple which accept ByteBuffer and try to deserialize it. >> >> following is the code in the thrift server api >> >> DatumReader reader = new >> GenericDatumReader(IMAGE_SCHEMA_1); >> ByteArrayOutputStream out = new ByteArrayOutputStream(); >> try { >> out.write(bf.array()); >> } catch (IOException e1) { >> // TODO Auto-generated catch block >> e1.printStackTrace(); >> } >> BinaryDecoder decoder = >> DecoderFactory.get().binaryDecoder(out.toByteArray(), null); >> try { >> GenericRecord result = reader.read(null, decoder); >> System.out.println("RESULT : " + result.toString()); >> } catch (IOException e) { >> // TODO Auto-generated catch block >> e.printStackTrace(); >> } >> >> >> >> What is wrong here. >> >> >> Thanks >> Aviansh >> >> >> > --001a11c2012ccc069604df531d39 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
It's hard to tell just by looking at the code -- I sug= gest you inspect the contents of the array you're decoding in each case= , and see if there is any difference.

Purely guess= ing, the ByteBuffer that Thrift gives you may have other data before and af= ter the Avro record, and calling `bf.array()` will give you junk in that ca= se. You may need something more like this (rough outline, please read the A= PI docs):

byte[] copy =3D new byte[bf.limit() - bf.po= sition()];
bf.read(copy);
DecoderFactory.get().binaryDecoder(copy<= /span>, null);<= /span>



On 16 June 2013 20:15, Avinash Dongre <dongre.a= vinash@gmail.com> wrote:

Hi,
I am trying to send avro serialized data to my thr= ift server. But when I try to de-serialized I get exception=A0
org.apache.avro.AvroRuntimeException: Malformed data. Length is n= egative: -51

If I do the same operation in the same vm process= everything works fine.

Here is my small code.

public static GenericRecord createContentNestedO= bject() throws Exception {
=A0 =A0 GenericRecord image1 =3D new GenericData.Record(IMAGE_SCHEMA);=
=A0 =A0 image1.put("uri", new Utf8("http://javaone.com/ke= ynote_large.jpg"));
=A0 =A0 image1.put("width", 0);
=A0 =A0 image1.put= ("height", 0);
=A0 =A0 image1.put("size", 2);=
=A0 =A0 image1.put("title", new Utf8("Javaone Key= note"));
=A0 =A0 return image1;
=A0 }

=A0 //= Helper Method to serialize the object from avro to bytebuffer
= =A0 public static ByteBuffer serialize(GenericRecord content, Schema schema= )
=A0 =A0 =A0 throws Exception {

=A0 =A0 ByteArrayOu= tputStream out =3D new ByteArrayOutputStream();
=A0 =A0 DatumWrit= er<GenericRecord> writer =3D new GenericDatumWriter<GenericRecord&= gt;(
=A0 =A0 =A0 =A0 schema);
=A0 =A0 // create Binary Encoder
=A0 =A0 EncoderFactory ef =3D new EncoderFactory();
=A0 =A0 Bi= naryEncoder be =3D ef.binaryEncoder(out, null);
=A0 =A0 writer.wr= ite(content, be);
=A0 =A0 be.flush();
=A0 =A0 out.close();

=A0 =A0 return ByteBuffer.wrap(out.toByteArray());
=A0 }

=A0 public static void main(String[] args) throws Excepti= on {
=A0 =A0 try {
=A0 =A0 =A0 GenericRecord rd =3D createContentN= estedObject();
=A0 =A0 =A0 bf =3D serialize(rd, IMAGE_SCHEMA);
=A0 =A0 =A0=A0
=A0 =A0 } catch (Exception e) {
<= div>=A0 =A0 =A0 // TODO Auto-generated catch block
=A0 =A0 =A0 e.printStackTrace();
=A0 =A0 }
=A0 =A0= =A0
=A0 =A0 // Deserialization.=A0
=A0 =A0 DatumReader&= lt;GenericRecord> reader =3D new GenericDatumReader<GenericRecord>= (IMAGE_SCHEMA_1); =A0 =A0
=A0 =A0 ByteArrayOutputStream out =3D new ByteArrayOutputStream(= );
=A0 =A0 try {
=A0 =A0 =A0 out.write(bf.array());
=A0 =A0 } catch (IOException e1) {
=A0 =A0 =A0 // TODO Aut= o-generated catch block
=A0 =A0 =A0 e1.printStackTrace();
=A0 =A0 } =A0 =A0
=A0 =A0 BinaryDecoder decoder =3D DecoderFactory.get().binaryDecoder(out.= toByteArray(), null); =A0
=A0 =A0 try {
=A0 =A0 =A0 Gen= ericRecord result =3D reader.read(null, decoder);
=A0 =A0 =A0 System.out.println("RESULT : " + result.toString= ());
=A0 =A0 } catch (IOException e) {
=A0 =A0 =A0 // T= ODO Auto-generated catch block
=A0 =A0 =A0 e.printStackTrace();
=A0 =A0 }

This code works. When I send the bf to my thrift api , = I am getting the error,=A0
my thrift api is simple which accept B= yteBuffer and try to deserialize it.

following is the code in the thrift server api

DatumReader<GenericRecord> reader =3D new GenericDatumReader&l= t;GenericRecord>(IMAGE_SCHEMA_1); =A0 =A0
=A0 =A0 ByteArrayOutputStream out =3D new ByteArrayOutputStream();
=A0 =A0 try {
=A0 =A0 =A0 out.write(bf.array());
=A0 = =A0 } catch (IOException e1) {
=A0 =A0 =A0 // TODO Auto-generated= catch block
=A0 =A0 =A0 e1.printStackTrace();
=A0 =A0 } =A0 =A0
=A0= =A0 BinaryDecoder decoder =3D DecoderFactory.get().binaryDecoder(out.toByt= eArray(), null); =A0
=A0 =A0 try {
=A0 =A0 =A0 GenericR= ecord result =3D reader.read(null, decoder);
=A0 =A0 =A0 System.out.println("RESULT : " + result.toString= ());
=A0 =A0 } catch (IOException e) {
=A0 =A0 =A0 // T= ODO Auto-generated catch block
=A0 =A0 =A0 e.printStackTrace();
=A0 =A0 }



What is wrong here.
=


Thanks
Aviansh

<= /div>



--001a11c2012ccc069604df531d39--