Return-Path: Delivered-To: apmail-hadoop-avro-user-archive@minotaur.apache.org Received: (qmail 44408 invoked from network); 6 Aug 2009 23:53:44 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 6 Aug 2009 23:53:44 -0000 Received: (qmail 1239 invoked by uid 500); 6 Aug 2009 23:53:51 -0000 Delivered-To: apmail-hadoop-avro-user-archive@hadoop.apache.org Received: (qmail 1171 invoked by uid 500); 6 Aug 2009 23:53:51 -0000 Mailing-List: contact avro-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: avro-user@hadoop.apache.org Delivered-To: mailing list avro-user@hadoop.apache.org Received: (qmail 1162 invoked by uid 99); 6 Aug 2009 23:53:51 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 23:53:51 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of tazan007@gmail.com designates 74.125.92.26 as permitted sender) Received: from [74.125.92.26] (HELO qw-out-2122.google.com) (74.125.92.26) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 23:53:41 +0000 Received: by qw-out-2122.google.com with SMTP id 8so483980qwh.35 for ; Thu, 06 Aug 2009 16:53:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=zrpd+RmTs++EwS+deb7s0w+krLcbID4eLA7/ly6VcWM=; b=eDUIVmgu65oAmxFTVYcD2Ji2A19NMJbpwQQyiKHCGFPNvAIphOsKFWVz8WIT+EqV+I H1My3HNUfflK67BsW1nhoukpBaIMw9tkB8zZb0FcuTcRGTGFk7VkYm6SL73hrQ/euy0a 4qvMfBIPH+7lMqmWknBxLWrAKwaNaUR1t3d7Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=SBBgP5iSCvVy4U/n0n8cxnrvXZvnK5piZ0cMWcuk4lt13KzvDW5ZbGL8jkegaUZOWy AFPYwizHtOAY55hVMid9sX8VQNTiPTp7ArN1TyyE/a2Irbm1oD90eZL7IT6GwVsDe4++ LmWuF8lwWrhIXper1yuvtX5juCQw3qD0sWm5I= MIME-Version: 1.0 Received: by 10.229.74.71 with SMTP id t7mr414601qcj.67.1249602800377; Thu, 06 Aug 2009 16:53:20 -0700 (PDT) In-Reply-To: <4A7B4ABF.5020201@apache.org> References: <1ba4850e0907311551i918d193qc4f75a3812dbcedd@mail.gmail.com> <4A738497.3010407@apache.org> <1ba4850e0907311705p66ea4219t9ffb59a33fea9972@mail.gmail.com> <4A7B4ABF.5020201@apache.org> Date: Thu, 6 Aug 2009 16:53:20 -0700 Message-ID: <1ba4850e0908061653t6f3654ffsfba7d3670c5b3af7@mail.gmail.com> Subject: Re: Unknown type java.util.List with ReflectData.getSchema From: tazan007 To: avro-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=00163642708406e613047081d36f X-Virus-Checked: Checked by ClamAV on apache.org --00163642708406e613047081d36f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Yea makes sense. I am trying to go from Java to Avro :D so I ended up overriding getArraySize, getArrayElements, writeRecord, and writeString from ReflectDatumWriter so I could have it convert String to Utf8, Date to Long, and List to GenericArray. Also had to make some changes to ReflectData's createSchema to support String and Lists and changed RecordSchema's fieldsToJson to translate Date objects to long. Basically a cluge but works for getting data out of my Java objects into Avro objects. If anyone is interested in doing the same, let me know. -Hiral On Thu, Aug 6, 2009 at 2:27 PM, Doug Cutting wrote: > Sorry, List is not supported, rather one must use > GenericArray. The reason is that both the specific and the reflect > implementations inherit from the generic implementation, primarily just > overriding the implementation of records. So the mapping from Avro to Java > (excepting records) is described at: > > > http://hadoop.apache.org/avro/docs/current/api/java/org/apache/avro/generic/package-summary.html > > The GenericArray interface is mostly a subset of the List interface, but > includes one non-List method, peek(), in order to permit reuse of instances. > > Doug > > tazan007 wrote: > >> Still getting similar error with Utf8: >> >> ptype = java.util.List raw = interface >> java.util.List >> param =class org.apache.avro.util.Utf8 >> Exception in thread "main" org.apache.avro.AvroTypeException: Unknown >> type: java.util.List >> at >> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:192) >> at >> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:185) >> at org.apache.avro.reflect.ReflectData.getMessage(ReflectData.java:230) >> at >> org.apache.avro.reflect.ReflectData.getProtocol(ReflectData.java:206) >> >> Are lists not support? >> >> -Hiral >> >> On Fri, Jul 31, 2009 at 4:56 PM, Doug Cutting > cutting@apache.org>> wrote: >> >> Avro uses org.apache.avro.util.Utf8 to represent strings, not >> java.lang.String. >> >> Doug >> >> >> tazan007 wrote: >> >> Hi guys, I am getting this error when trying to build a schema >> using an existing class with ReflectData.getSchema method with a >> List field. Is this expected behaviour? Are List's not >> supported? >> >> ptype = java.util.List raw = interface >> java.util.List >> param =class java.lang.String >> Exception in thread "main" org.apache.avro.AvroTypeException: >> Unknown type: java.util.List >> at >> >> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:192) >> at >> >> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:185) >> at >> org.apache.avro.reflect.ReflectData.getSchema(ReflectData.java:107) >> >> Thanks, >> Hiral >> >> >> --00163642708406e613047081d36f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Yea makes sense.=C2=A0 I am trying to go from Java to Avro :D so I ended up= overriding getArraySize, getArrayElements, writeRecord, and writeString fr= om ReflectDatumWriter so I could have it convert String to Utf8, Date to Lo= ng, and List to GenericArray.=C2=A0

Also had to make some changes to ReflectData's createSchema to supp= ort String and Lists=C2=A0 and changed RecordSchema's fieldsToJson to t= ranslate Date objects to long.=C2=A0 Basically a cluge but works for gettin= g data out of my Java objects into Avro objects.

If anyone is interested in doing the same, let me know.

-Hiral<= br>
On Thu, Aug 6, 2009 at 2:27 PM, Doug Cutt= ing <cutting@apa= che.org> wrote:
Sorry, List<St= ring> is not supported, rather one must use GenericArray<Utf8>. = =C2=A0The reason is that both the specific and the reflect implementations = inherit from the generic implementation, primarily just overriding the impl= ementation of records. =C2=A0So the mapping from Avro to Java (excepting re= cords) is described at:

http://hadoop.apache.or= g/avro/docs/current/api/java/org/apache/avro/generic/package-summary.html

The GenericArray interface is mostly a subset of the List interface, but in= cludes one non-List method, peek(), in order to permit reuse of instances.<= br>
Doug

tazan007 wrote:
Still getting similar error with Utf8:

ptype =3D java.util.List<org.apache.avro.util.Utf8> raw =3D interface= java.util.List
param =3Dclass org.apache.avro.util.Utf8
Exception in thread "main" org.apache.avro.AvroTypeException: Unk= nown type: java.util.List<org.apache.avro.util.Utf8>
=C2=A0 =C2=A0at org.apache.avro.reflect.ReflectData.createSchema(ReflectDa= ta.java:192)
=C2=A0 =C2=A0at org.apache.avro.reflect.ReflectData.createSchema(ReflectDa= ta.java:185)
=C2=A0 =C2=A0at org.apache.avro.reflect.ReflectData.getMessage(ReflectData= .java:230)
=C2=A0 =C2=A0at org.apache.avro.reflect.ReflectData.getProtocol(ReflectDat= a.java:206)

Are lists not support?

-Hiral

On Fri, Jul 31, 2009 at 4:56 PM, Doug Cutting <cutting@apache.org <mailto:cutting@apache.org>>= ; wrote:

=C2=A0 =C2=A0Avro uses org.apache.avro.util.Utf8 to represent strings, not=
=C2=A0 =C2=A0java.lang.String.

=C2=A0 =C2=A0Doug


=C2=A0 =C2=A0tazan007 wrote:

=C2=A0 =C2=A0 =C2=A0 =C2=A0Hi guys, I am getting this error when trying to= build a schema
=C2=A0 =C2=A0 =C2=A0 =C2=A0using an existing class with ReflectData.getSch= ema method with a
=C2=A0 =C2=A0 =C2=A0 =C2=A0List<String> field. =C2=A0Is this expecte= d behaviour? =C2=A0Are List's not
=C2=A0 =C2=A0 =C2=A0 =C2=A0supported?

=C2=A0 =C2=A0 =C2=A0 =C2=A0ptype =3D java.util.List<java.lang.String>= ; raw =3D interface
=C2=A0 =C2=A0 =C2=A0 =C2=A0java.util.List
=C2=A0 =C2=A0 =C2=A0 =C2=A0param =3Dclass java.lang.String
=C2=A0 =C2=A0 =C2=A0 =C2=A0Exception in thread "main" org.apache= .avro.AvroTypeException:
=C2=A0 =C2=A0 =C2=A0 =C2=A0Unknown type: java.util.List<java.lang.Strin= g>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at
=C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.avro.reflect.ReflectData.createSchem= a(ReflectData.java:192)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at
=C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.avro.reflect.ReflectData.createSchem= a(ReflectData.java:185)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 at
=C2=A0 =C2=A0 =C2=A0 =C2=A0org.apache.avro.reflect.ReflectData.getSchema(R= eflectData.java:107)

=C2=A0 =C2=A0 =C2=A0 =C2=A0Thanks,
=C2=A0 =C2=A0 =C2=A0 =C2=A0Hiral



--00163642708406e613047081d36f--