Return-Path: Delivered-To: apmail-hadoop-avro-user-archive@minotaur.apache.org Received: (qmail 6342 invoked from network); 1 Dec 2009 20:56:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 1 Dec 2009 20:56:01 -0000 Received: (qmail 28219 invoked by uid 500); 1 Dec 2009 20:56:01 -0000 Delivered-To: apmail-hadoop-avro-user-archive@hadoop.apache.org Received: (qmail 28179 invoked by uid 500); 1 Dec 2009 20:56:01 -0000 Mailing-List: contact avro-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: avro-user@hadoop.apache.org Delivered-To: mailing list avro-user@hadoop.apache.org Received: (qmail 28170 invoked by uid 99); 1 Dec 2009 20:56:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2009 20:56:01 +0000 X-ASF-Spam-Status: No, hits=-2.6 required=5.0 tests=AWL,BAYES_00,HTML_MESSAGE X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [12.11.0.150] (HELO pssd.com) (12.11.0.150) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2009 20:55:59 +0000 Received: from ([10.10.0.123]) by wfldspam01.pssd.com with ESMTP id 9HPHRF1.172153624; Tue, 01 Dec 2009 15:52:00 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CA72C8.8350CAC1" Subject: RE: reflected char array returning null package. Date: Tue, 1 Dec 2009 15:54:45 -0500 Message-ID: <33FF7983173AFC4491EE3C207E85231C3F2D39@WFLDEXC14.main.pssworldmedical.com> In-Reply-To: <1ba4850e0912011156m35d38b2aj735d0ba085c736df@mail.gmail.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: reflected char array returning null package. Thread-Index: AcpywH4ppAVADew0TROH9SF313HYkQAAErZQ References: <33FF7983173AFC4491EE3C207E85231C3F2D1D@WFLDEXC14.main.pssworldmedical.com> <4B155EDE.1070400@apache.org> <33FF7983173AFC4491EE3C207E85231C3F2D35@WFLDEXC14.main.pssworldmedical.com> <1ba4850e0912011156m35d38b2aj735d0ba085c736df@mail.gmail.com> From: "Marko Milicevic" To: This is a multi-part message in MIME format. ------_=_NextPart_001_01CA72C8.8350CAC1 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks for the offer Hiral, but without doc i'm not sure what i need. =20 My original usecase was to serialize arbitrary Java objects over the wire. I was looking for something more efficient than Java serialization. =20 I read of several benchmarks that showed favorable performance for Avro, so here i am. =20 But it seems that there are significant constraints on the type of Java object that can be serialized. If that is the case then i tend to think that Doug's approach would make the most sense in the long run (ie. first define a Avro schema/protocol, then generate the Java). =20 Thanks again. =20 Marko. . ________________________________ From: tazan007 [mailto:tazan007@gmail.com]=20 Sent: Tuesday, December 01, 2009 2:57 PM To: avro-user@hadoop.apache.org Subject: Re: reflected char array returning null package. Marko, I did make some changes to avro 1.0 back when I was evaluating it for RPC and added support for it to convert String to Utf8 and Date to long for converting Java to Avro but I never had to go from Avro back to Java once I had the data in Avro. If you want, I could try to apply the changes I made to 1.2 if it fits your use-case. -Hiral On Tue, Dec 1, 2009 at 11:42 AM, Marko Milicevic wrote: Thanks Doug. =09 Is there any documentation on the Specific API and the Specific schema format? =09 Marko. . =09 =09 -----Original Message----- From: Doug Cutting [mailto:cutting@apache.org] Sent: Tuesday, December 01, 2009 1:22 PM To: avro-user@hadoop.apache.org Subject: Re: reflected char array returning null package. =09 =09 In Avro 1.2's reflect implementation, Strings must be represented with Avro's Utf8 class. Folks reasonably complained that this is not very useful, and it's being fixed for Avro 1.3. =09 https://issues.apache.org/jira/browse/AVRO-80 =09 I'm currently investing a lot of effort improving reflect, and reflect should work considerably better in 1.3. That said, unless you have an existing RPC codebase that you need to move onto Avro with minimal disruption, the specific API is much easier to use than the reflect API. =09 Avro schemas are mostly an expressive subset of Java, so mapping Avro to Java is much simpler than mapping Java to Avro. =09 Doug =09 Marko Milicevic wrote: > Hello. I'm running the following code under the avro 1.2.0 release > and jdk 1.6.0_10. > > public interface ISink > { > public void send( String msg ); > } > > public class Sink implements ISink > { > public void send( String msg ) > { > } > } > > > public void test() > { > SocketServer server =3D > new SocketServer(new SpecificResponder( > ISink.class, new Sink()), new InetSocketAddress(3333)); > > SocketTransceiver client =3D > new SocketTransceiver(new InetSocketAddress(server.getPort())); > > ISink proxy =3D (ISink)SpecificRequestor.getClient(ISink.class, > client); > > for (int i =3D 0; i < 1000; i++ ) > { > proxy.send("testing"); > } > > client.close(); > server.close(); > } > > > First I was getting errors until I learned of paranamer. > I thought I read in the doc, that no pre-processing would be necessary? > > I'm fine with having to run paranamer, but want to be sure that it is > required? > > > Now after processing with paranamer I get the following error trying > to run... > > Exception in thread "main" java.lang.NullPointerException > at > org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:259) > at > org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java > :3 > 03) > at > org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:291) > at > org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java > :3 > 03) > at > org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:291) > at > org.apache.avro.reflect.ReflectData.getMessage(ReflectData.java:339) > at > org.apache.avro.reflect.ReflectData.getProtocol(ReflectData.java:318) > at > org.apache.avro.reflect.ReflectResponder.(ReflectResponder.java: > 45 > ) > at > org.apache.avro.reflect.ReflectResponder.(ReflectResponder.java: > 41 > ) > > // > org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:259) > } else if (type instanceof Class) { > Class c =3D (Class)type; > String name =3D c.getSimpleName(); >>> String space =3D c.getPackage().getName(); > if (c.getEnclosingClass() !=3D null) // nested class > space =3D c.getEnclosingClass().getName() + "$"; > String fullName =3D c.getName(); > Schema schema =3D names.get(fullName); > if (schema =3D=3D null) { > > Stepping though the debugger, the type appears to be "class [C", with > the name "value". I assume that is the "value" char[] in the String > msg. > > Should a Class of type "class [C", return a package of null? > > If so, any idea what I'm doing incorrectly? > > Thank you. > > Marko. > . =09 ------_=_NextPart_001_01CA72C8.8350CAC1 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Thanks for the offer Hiral, but without doc i'm = not sure=20 what i need.
 
My original usecase was to serialize arbitrary = Java objects=20 over the wire.  I was looking for something more efficient than = Java=20 serialization.
 
I read of several benchmarks that showed = favorable=20 performance for Avro, so here i am.
 
But it seems that there are significant = constraints on the=20 type of Java object that can be serialized.  If that is the case = then i=20 tend to think that Doug's approach would make the most sense in the long = run=20 (ie. first define a Avro schema/protocol, then generate the=20 Java).
 
Thanks again.
 
Marko.
.

From: tazan007 = [mailto:tazan007@gmail.com]=20
Sent: Tuesday, December 01, 2009 2:57 PM
To:=20 avro-user@hadoop.apache.org
Subject: Re: reflected char array=20 returning null package.

Marko, I did make some changes to avro 1.0 back when I was = evaluating=20 it for RPC and added support for it to convert String to Utf8 and Date = to long=20 for converting Java to Avro but I never had to go from Avro back to Java = once I=20 had the data in Avro.

If you want, I could try to apply the = changes I=20 made to 1.2 if it fits your use-case.

-Hiral

On Tue, Dec 1, 2009 at 11:42 AM, Marko = Milicevic <MMilicevic@pssd.com> = wrote:
Thanks=20 Doug.

Is there any documentation on the Specific API and the = Specific=20 schema
format?

Marko.
.

-----Original Message-----
From: Doug Cutting = [mailto:cutting@apache.org]
Sent: = Tuesday,=20 December 01, 2009 1:22 PM
To: avro-user@hadoop.apache.org
Subject:=20 Re: reflected char array returning null package.

In Avro 1.2's reflect implementation, Strings must be=20 represented with
Avro's Utf8 class.  Folks reasonably = complained that=20 this is not very
useful, and it's being fixed for Avro = 1.3.

https://issues.apache.org/jira/browse/AVRO-80

= I'm=20 currently investing a lot of effort improving reflect, and = reflect
should=20 work considerably better in 1.3.  That said, unless you have=20 an
existing RPC codebase that you need to move onto Avro with=20 minimal
disruption, the specific API is much easier to use than the = reflect=20 API.

 Avro schemas are mostly an expressive subset of = Java, so=20 mapping Avro
to Java is much simpler than mapping Java to=20 Avro.

Doug

Marko Milicevic wrote:
> Hello. =  I'm=20 running the following code under the avro 1.2.0 release
> and = jdk=20 1.6.0_10.
>
> public interface ISink
> {
> =  =20   public void send( String msg );
> }
>
> = public class=20 Sink implements ISink
> {
>       public = void send(=20 String msg )
>       {
>     =  =20 }
> }
>
>
> public void test()
> = {
>=20     SocketServer server =3D
>       new = SocketServer(new SpecificResponder(
>       =    =20   ISink.class, new Sink()), new = InetSocketAddress(3333));
>
>=20     SocketTransceiver client =3D
>     =   new=20 SocketTransceiver(new = InetSocketAddress(server.getPort()));
>
>=20     ISink proxy =3D=20 (ISink)SpecificRequestor.getClient(ISink.class,
>=20 client);
>
>     for (int i =3D 0; i < 1000; = i++=20 )
>     {
>        =20 proxy.send("testing");
>     }
>
>   =  =20 client.close();
>     server.close();
>=20 }
>
>
> First I was getting errors until I learned = of=20 paranamer.
> I thought I read in the doc, that no pre-processing = would=20 be
necessary?
>
> I'm fine with having to run = paranamer, but=20 want to be sure that it is
> required?
>
>
> = Now after=20 processing with paranamer I get the following error trying
> to=20 run...
>
> Exception in thread "main"=20 java.lang.NullPointerException
>       at
> = = org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:259)>=20       at
>=20 = org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java>=20 :3
> 03)
>       at
>=20 = org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:291)>=20       at
>=20 = org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java>=20 :3
> 03)
>       at
>=20 = org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:291)>=20       at
>=20 = org.apache.avro.reflect.ReflectData.getMessage(ReflectData.java:339)
&= gt;=20       at
>=20 = org.apache.avro.reflect.ReflectData.getProtocol(ReflectData.java:318)
= >=20       at
>=20 = org.apache.avro.reflect.ReflectResponder.<init>(ReflectResponder.ja= va:
>=20 45
> )
>       at
>=20 = org.apache.avro.reflect.ReflectResponder.<init>(ReflectResponder.ja= va:
>=20 41
> )
>
> //
>=20 = org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:259)>=20   } else if (type instanceof Class) {
>     =   Class=20 c =3D (Class)type;
>       String name =3D=20 c.getSimpleName();
>>>      String space = =3D=20 c.getPackage().getName();
>       if=20 (c.getEnclosingClass() !=3D null)          // = nested=20 class
>         space =3D=20 c.getEnclosingClass().getName() + "$";
>       = String=20 fullName =3D c.getName();
>       Schema schema = =3D=20 names.get(fullName);
>       if (schema =3D=3D = null)=20 {
>
> Stepping though the debugger, the type appears to be = "class=20 [C", with
> the name "value".  I assume that is the "value" = char[]=20 in the String
> msg.
>
> Should a Class of type = "class [C",=20 return a package of null?
>
> If so, any idea what I'm = doing=20 incorrectly?
>
> Thank you.
>
> Marko.
> = .

------_=_NextPart_001_01CA72C8.8350CAC1--