Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B8112200BFF for ; Tue, 17 Jan 2017 16:11:52 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id B318F160B43; Tue, 17 Jan 2017 15:11:52 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 69180160B46 for ; Tue, 17 Jan 2017 16:11:51 +0100 (CET) Received: (qmail 65229 invoked by uid 500); 17 Jan 2017 15:11:50 -0000 Mailing-List: contact user-help@avro.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@avro.apache.org Delivered-To: mailing list user@avro.apache.org Received: (qmail 65216 invoked by uid 99); 17 Jan 2017 15:11:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Jan 2017 15:11:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D5E1AC1264 for ; Tue, 17 Jan 2017 15:11:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.389 X-Spam-Level: ** X-Spam-Status: No, score=2.389 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=intentmedia.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 9ZahPSv4230R for ; Tue, 17 Jan 2017 15:11:48 +0000 (UTC) Received: from mail-yw0-f181.google.com (mail-yw0-f181.google.com [209.85.161.181]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 08B245FB0F for ; Tue, 17 Jan 2017 15:11:48 +0000 (UTC) Received: by mail-yw0-f181.google.com with SMTP id l19so89588444ywc.2 for ; Tue, 17 Jan 2017 07:11:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intentmedia.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=+L68fjoOsQE1AjKfcAFHf6qdelD7nijCmMy2xKTsNR0=; b=X6ft3K00oXZeD2S/CMioj2VUyLR13yfwte8bXS/AOetWYQq6o97pwSeafzNb5aFmjy fvGf8XFPk0JZj5R4C91DXR/FZZbP6P3YOUxwT4TzImKWTVjzPyqWWSXFxUm/gsut7luD CoSB+aEjAPbiDZeVsyVyvma4uBwGRwVaOCj/s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=+L68fjoOsQE1AjKfcAFHf6qdelD7nijCmMy2xKTsNR0=; b=DuqoFlFTbO/BRb9G6bL6Xhw9OYgtN7EnPjProfhxf7Ho6NDo/m+zQRShDwUfZgAYYW p0ov1JSQ4TKEGQml/qhnuLhPub7EDmPIIm396wlAcAmpXcg/09GGwa8t9oujK77k8WWa OFn9PQ+8oD2ny13NAN1EQGltjUHqhK141iu+F9w+rLMqyIuiBmlGdwcrEYO6vdDYepFn AtIcmHpcHLNJXJzSVCCaKMgzhQnoaU1iVlp4s1V1wGjhFNd8AYiWGJPmG+oO5bzMBrbr x7hIMEWzCuTq8beekvkNEU/KNXiTln34rnU0kcKDe1z50GgxZitbmrhZPyj8zrIn+MPV S9kQ== X-Gm-Message-State: AIkVDXLLHzMGGpyBzbeRt/9judesmHbl44k1X9YpgkHw/leqljI+7k9ddhP2WGTUzm3Mc1wFyfii51Tj6BVZSfIg X-Received: by 10.55.27.65 with SMTP id b62mr38764007qkb.202.1484665906542; Tue, 17 Jan 2017 07:11:46 -0800 (PST) MIME-Version: 1.0 Received: by 10.140.82.20 with HTTP; Tue, 17 Jan 2017 07:11:46 -0800 (PST) In-Reply-To: References: From: Jeremy Fine Date: Tue, 17 Jan 2017 10:11:46 -0500 Message-ID: Subject: Re: When using the SpecificWriter with .net, I get an exception when a referenced object in my schema is null To: user@avro.apache.org Content-Type: multipart/alternative; boundary=001a1147f044f87dec05464bb7ab archived-at: Tue, 17 Jan 2017 15:11:52 -0000 --001a1147f044f87dec05464bb7ab Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable does avdl this work? i think you need to specify that a field can be null. @namespace("sample") protocol SampleAvro { record Sample { union { null, string } foo; union { null, Sample } bar; } } On Mon, Jan 16, 2017 at 1:01 PM, Rabe, Jens wrote: > Hello again, > > > > it turns out I was wrong in first place. I re-tested it in Scala and get > the same error. > > > > Gues I=E2=80=99ll have to redesign the schema then to avoid the nulls. > > > > *Von:* Rabe, Jens [mailto:jens.rabe@iwes.fraunhofer.de] > *Gesendet:* Montag, 16. Januar 2017 18:13 > *An:* user@avro.apache.org > *Betreff:* When using the SpecificWriter with .net, I get an exception > when a referenced object in my schema is null > > > > Hello Avro community, > > > > I want to use my Avro serialized data with .net and I stumbled upon a > strange problem. When a record in my schema references another record, an= d > I set it to null, when serializing this in .net this throws an exception. > > > > Here is what I did: > > > > First, I created a simple protocol. This is my avdl file: > > > > @namespace("sample") > > > > protocol SampleAvro { > > > > record Sample { > > > > string foo; > > Sample bar; > > } > > } > > > > Now, I generated C# classes out of it: > > > > java -jar avro-tools-1.8.0.jar idl sample.avdl >sample.json > > avrogen -p sample.json . > > > > I then copied the .cs files into a project, and tried to serialize a > sample object where the =E2=80=9Cbar=E2=80=9D reference is null: > > > > using(var ms =3D new MemoryStream()) { > > var sw =3D new SpecificWriter(Sample._SCHEMA); > > sw.Write(new Sample { foo =3D "bar" }, new BinaryEncoder(ms))= ; > > } > > > > I get the following exception: *Record object is not derived from > ISpecificRecord in field bar* > > > > When looking at the source code, this happens in the SpecificWriter.cs in > the following method (I made the relevant lines bold): > > > > protected override void WriteRecord(RecordSchema schema, object v= alue, Encoder encoder) > > { > > * var rec =3D value as ISpecificRecord;* > > * if (rec =3D=3D null)* > > * throw new AvroTypeException("Record object is not derive= d from ISpecificRecord");* > > > > foreach (Field field in schema) > > { > > try > > { > > Write(field.Schema, rec.Get(field.Pos), encoder); > > } > > catch (Exception ex) > > { > > throw new AvroException(ex.Message + " in field " + f= ield.Name); > > } > > } > > } > > > > The =E2=80=9Cvalue as ISpecificRecord=E2=80=9D returns null here because = the reference is > null. This is intended in my schema, but throws an exception here. > > > > Is this a bug or is this intended behavior, and either way, how should I > work around this? I need situations where the =E2=80=9Cbar=E2=80=9D is nu= ll, and on the > JVM/Scala side this is no problem and working flawlessly. > > > > Regards, > > Jens > --=20 [image: Intent Media] *Jeremy Fine* Software Engineer 315 Hudson Street, 9th Floor, New York, NY 10013 We've been named a Crain's "Best Place to Work" for three years running! --001a1147f044f87dec05464bb7ab Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable 
does avdl th=
is work? i think you need to specify that a field can be null.

@names=
pace("sample")
protocol Sampl=
eAvro {
=C2=A0
record Sample {
=C2=A0
=

=C2=A0=C2=A0union { null, string }=
=C2=A0foo;
=C2=A0=C2=A0union { null, Sample }=C2=A0bar;
}
}



On Mon, Jan 16, 2017 at 1:01 PM, Rabe, Jens <jens.rabe@iwes.fraunhofer.de> wrote:

Hello again,=

=C2=A0

it turn= s out I was wrong in first place. I re-tested it in Scala and get the same = error.

= =C2=A0

Gues I= =E2=80=99ll have to redesign the schema then to avoid the nulls.<= /u>

= =C2=A0

Von: Rabe, Jen= s [mailto:jens.rabe@iwes.fraunhofer.de]
Gesendet: Montag, 16. Januar 2017 18:13
An: user@a= vro.apache.org
Betreff: When using the SpecificWriter with .net, I get an exception= when a referenced object in my schema is null

=C2=A0

Hello Avro community,=

=C2=A0

I want to use my Avro serialize= d data with .net and I stumbled upon a strange problem. When a record in my= schema references another record, and I set it to null, when serializing t= his in .net this throws an exception.

=C2=A0

Here is what I did:

=C2=A0

First, I created a simple proto= col. This is my avdl file:

=C2=A0

@namespace("sample")

=C2=A0

protocol SampleAvro {<= /p>

=C2=A0

record Sample {

=C2=A0

=C2=A0 string foo;

=C2=A0 Sample bar;

}

}

=C2=A0

Now, I generated C# classes out= of it:

=C2=A0

java -<=
/span>=
jar avro-tools-=
1.8.0.jar idl sample.avdl >sample.json
avrogen -p sample.json .

=C2=A0

I then copied the .cs files int= o a project, and tried to serialize a sample object where the =E2=80=9Cbar= =E2=80=9D reference is null:

=C2=A0

=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 using(var ms =3D =
new <=
/span>MemoryS=
tream()) {
=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
var sw =3D<=
/span>=
 new SpecificWriter<Sample>(Sample._SCHEMA);<=
u>
=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sw.Write(n=
ew Sample { foo =3D "bar" }, new BinaryEncoder(<=
/span>=
ms));
=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }

=C2=A0

I get the following exception: = Record object is not derived from ISpecificRecord in field bar

=C2=A0

When looking at the source code= , this happens in the SpecificWriter.cs in the following method (I made the= relevant lines bold):

=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 protec=
ted override void WriteRecord(RecordSchema schema, object value, Encoder en=
coder)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 {
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0 var rec =3D value as ISpecificRecord;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0 if (rec =3D=3D null)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 throw new AvroTypeException(&=
quot;Record object is not derived from ISpecificRecord");
=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 foreach (Field field in schema)

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 {


=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 try


=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 {


=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Write(fi=
eld.Schema, rec.Get(field.Pos), encoder);


=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }


=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 catch (Exception ex)


=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 {


=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 throw ne=
w AvroException(ex.Message + " in field " + field.Name);


=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }


=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 }


=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }


=C2=A0


The =E2=80=9Cvalue as ISpecific=
Record=E2=80=9D returns null here because the reference is null. This is in=
tended in my schema, but throws an exception here.


=C2=A0


Is this a bug or is this intend=
ed behavior, and either way, how should I work around this? I need situatio=
ns where the =E2=80=9Cbar=E2=80=9D is null, and on the JVM/Scala side this =
is no problem and working flawlessly.


=C2=A0


Regards,

Jens



--
--001a1147f044f87dec05464bb7ab--

3D"Intent

Je= remy Fine
Software Engineer=C2=A0
315 Hudson= Street, 9th Floor, New York, NY 10013
We've been named a Crain's "Be= st Place to Work" for three years running!