avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tazan007 <tazan...@gmail.com>
Subject Re: reflected char array returning null package.
Date Tue, 01 Dec 2009 21:10:22 GMT
Marko, no problem.  If you have the time, then defining your own
protocol/schema is the right approach.

I did apply my changes from 1.0 to 1.2 on my local git repo.  Here is a
simple use-case I tested with:

public class HelloWorld {
    public String m_strName="myname";
    public String m_strLocation="mylocation";
    public List<String> m_listString = new ArrayList<String>();
    public Date m_Date = new Date();
}

The above gets converted to the following schema using ReflectData:

{ "fields" : [ { "name" : "m_strName",
        "type" : "string"
      },
      { "name" : "m_strLocation",
        "type" : "string"
      },
      { "name" : "m_listString",
        "type" : { "items" : "string",
            "type" : "array"
          }
      },
      { "name" : "m_Date",
        "type" : { "fields" : [ { "name" : "data",
                  "type" : "long"
                } ],
            "name" : "Date",
            "namespace" : "java.util",
            "type" : "record"
          }
      }
    ],
  "name" : "HelloWorld",
  "namespace" : "com.company.project",
  "type" : "record"
}

Good luck with your Avro adventures :)

-Hiral

On Tue, Dec 1, 2009 at 12:54 PM, Marko Milicevic <MMilicevic@pssd.com>wrote:

>  Thanks for the offer Hiral, but without doc i'm not sure what i need.
>
> My original usecase was to serialize arbitrary Java objects over the wire.
> I was looking for something more efficient than Java serialization.
>
> I read of several benchmarks that showed favorable performance for Avro, so
> here i am.
>
> But it seems that there are significant constraints on the type of Java
> object that can be serialized.  If that is the case then i tend to think
> that Doug's approach would make the most sense in the long run (ie. first
> define a Avro schema/protocol, then generate the Java).
>
> Thanks again.
>
> Marko.
> .
>
>  ------------------------------
> *From:* tazan007 [mailto:tazan007@gmail.com]
> *Sent:* Tuesday, December 01, 2009 2:57 PM
>
> *To:* avro-user@hadoop.apache.org
> *Subject:* Re: reflected char array returning null package.
>
> Marko, I did make some changes to avro 1.0 back when I was evaluating it
> for RPC and added support for it to convert String to Utf8 and Date to long
> for converting Java to Avro but I never had to go from Avro back to Java
> once I had the data in Avro.
>
> If you want, I could try to apply the changes I made to 1.2 if it fits your
> use-case.
>
> -Hiral
>
> On Tue, Dec 1, 2009 at 11:42 AM, Marko Milicevic <MMilicevic@pssd.com>wrote:
>
>> Thanks Doug.
>>
>> Is there any documentation on the Specific API and the Specific schema
>> format?
>>
>> Marko.
>> .
>>
>> -----Original Message-----
>> From: Doug Cutting [mailto:cutting@apache.org]
>> Sent: Tuesday, December 01, 2009 1:22 PM
>> To: avro-user@hadoop.apache.org
>> Subject: Re: reflected char array returning null package.
>>
>>  In Avro 1.2's reflect implementation, Strings must be represented with
>> Avro's Utf8 class.  Folks reasonably complained that this is not very
>> useful, and it's being fixed for Avro 1.3.
>>
>> https://issues.apache.org/jira/browse/AVRO-80
>>
>> I'm currently investing a lot of effort improving reflect, and reflect
>> should work considerably better in 1.3.  That said, unless you have an
>> existing RPC codebase that you need to move onto Avro with minimal
>> disruption, the specific API is much easier to use than the reflect API.
>>
>>  Avro schemas are mostly an expressive subset of Java, so mapping Avro
>> to Java is much simpler than mapping Java to Avro.
>>
>> Doug
>>
>> Marko Milicevic wrote:
>> > Hello.  I'm running the following code under the avro 1.2.0 release
>> > and jdk 1.6.0_10.
>> >
>> > public interface ISink
>> > {
>> >     public void send( String msg );
>> > }
>> >
>> > public class Sink implements ISink
>> > {
>> >       public void send( String msg )
>> >       {
>> >       }
>> > }
>> >
>> >
>> > public void test()
>> > {
>> >     SocketServer server =
>> >       new SocketServer(new SpecificResponder(
>> >             ISink.class, new Sink()), new InetSocketAddress(3333));
>> >
>> >     SocketTransceiver client =
>> >       new SocketTransceiver(new InetSocketAddress(server.getPort()));
>> >
>> >     ISink proxy = (ISink)SpecificRequestor.getClient(ISink.class,
>> > client);
>> >
>> >     for (int i = 0; i < 1000; i++ )
>> >     {
>> >         proxy.send("testing");
>> >     }
>> >
>> >     client.close();
>> >     server.close();
>> > }
>> >
>> >
>> > First I was getting errors until I learned of paranamer.
>> > I thought I read in the doc, that no pre-processing would be
>> necessary?
>> >
>> > I'm fine with having to run paranamer, but want to be sure that it is
>> > required?
>> >
>> >
>> > Now after processing with paranamer I get the following error trying
>> > to run...
>> >
>> > Exception in thread "main" java.lang.NullPointerException
>> >       at
>> > org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:259)
>> >       at
>> > org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java
>> > :3
>> > 03)
>> >       at
>> > org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:291)
>> >       at
>> > org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java
>> > :3
>> > 03)
>> >       at
>> > org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:291)
>> >       at
>> > org.apache.avro.reflect.ReflectData.getMessage(ReflectData.java:339)
>> >       at
>> > org.apache.avro.reflect.ReflectData.getProtocol(ReflectData.java:318)
>> >       at
>> > org.apache.avro.reflect.ReflectResponder.<init>(ReflectResponder.java:
>> > 45
>> > )
>> >       at
>> > org.apache.avro.reflect.ReflectResponder.<init>(ReflectResponder.java:
>> > 41
>> > )
>> >
>> > //
>> > org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:259)
>> >   } else if (type instanceof Class) {
>> >       Class c = (Class)type;
>> >       String name = c.getSimpleName();
>> >>>      String space = c.getPackage().getName();
>> >       if (c.getEnclosingClass() != null)          // nested class
>> >         space = c.getEnclosingClass().getName() + "$";
>> >       String fullName = c.getName();
>> >       Schema schema = names.get(fullName);
>> >       if (schema == null) {
>> >
>> > Stepping though the debugger, the type appears to be "class [C", with
>> > the name "value".  I assume that is the "value" char[] in the String
>> > msg.
>> >
>> > Should a Class of type "class [C", return a package of null?
>> >
>> > If so, any idea what I'm doing incorrectly?
>> >
>> > Thank you.
>> >
>> > Marko.
>> > .
>>
>
>

Mime
View raw message