Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
MIME-Version: 1.0
References: 
 <CABbqrYnKHxJtCN_p_1reC=XDq88m_BssTKV97NX7nL9Yy5nf-Q@mail.gmail.com>
In-Reply-To: 
 <CABbqrYnKHxJtCN_p_1reC=XDq88m_BssTKV97NX7nL9Yy5nf-Q@mail.gmail.com>
From: David Ortiz <dpo5003@gmail.com>
Date: Thu, 06 Aug 2015 15:50:47 +0000
Message-ID: 
 <CAC79LcZVB_to1yHNCYddDWHoe=VcZo0bvZtCNXNDgm3Nt1qmTw@mail.gmail.com>
Subject: Re: ClassCastException: org.apache.avro.util.Utf8 cannot be cast to
 org.apache.avro.generic.IndexedRecord
To: Crunch users <user@crunch.apache.org>
Content-Type: multipart/alternative; boundary=001a113eca102c3046051ca67c4d

--001a113eca102c3046051ca67c4d
Content-Type: text/plain; charset=UTF-8

Believe the output type for a TextFileOutput has to be a String, not a
Table containing a String key with an Avro Map.  With the complex object I
think you have to write it out as an AvroFile.

On Thu, Aug 6, 2015 at 11:47 AM Hrishikesh P <hrishi.engineer@gmail.com>
wrote:

> I'm trying to write a PTable<String, Map<String, String>> to a
> TextFileSourceTarget but I am running into a ClassCastException.
>
> {code}
> PTable<String, Map<String, String>> table = ...;
>
> table.write(new
> TextFileSourceTarget<>(Path.mergePaths(fs.getHomeDirectory(), path),
> Avros.tableOf(Avros.strings(), Avros.maps(Avros.strings()))));
> {code}
>
> The exception is:
>
> WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to org.apache.avro.generic.IndexedRecord
> 	at org.apache.crunch.types.avro.AvroTableType$IndexedRecordToPair.map(AvroTableType.java:88)
> 	at org.apache.crunch.MapFn.process(MapFn.java:34)
> 	at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98)
> 	at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:109)
> 	at org.apache.crunch.impl.mr.run.CrunchMapper.map(CrunchMapper.java:60)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>
>
> Does someone have an idea why the exception might be occurring? The
> version of avro is 1.7.6. I have checked for dependency conflicts but there
> doesn't seem to be a problem with it although it looks like it.
>
> Thanks.
>

--001a113eca102c3046051ca67c4d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Believe the output type for a TextFileOutput has to be a S=
tring, not a Table containing a String key with an Avro Map.=C2=A0 With the=
 complex object I think you have to write it out as an AvroFile.</div><br><=
div class=3D"gmail_quote"><div dir=3D"ltr">On Thu, Aug 6, 2015 at 11:47 AM =
Hrishikesh P &lt;<a href=3D"mailto:hrishi.engineer@gmail.com">hrishi.engine=
er@gmail.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" styl=
e=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div di=
r=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:comic sans ms,s=
ans-serif;color:rgb(11,83,148)">I&#39;m trying to write a PTable&lt;String,=
 Map&lt;String, String&gt;&gt; to a TextFileSourceTarget but I am running i=
nto a ClassCastException.<br><br></div><div class=3D"gmail_default" style=
=3D"font-family:comic sans ms,sans-serif;color:rgb(11,83,148)">{code}<br>PT=
able&lt;String, Map&lt;String, String&gt;&gt; table =3D ...;<br><br></div><=
div class=3D"gmail_default" style=3D"font-family:comic sans ms,sans-serif;c=
olor:rgb(11,83,148)">table.write(new TextFileSourceTarget&lt;&gt;(Path.merg=
ePaths(fs.getHomeDirectory(), path), Avros.tableOf(Avros.strings(), Avros.m=
aps(Avros.strings()))));<br></div><div class=3D"gmail_default" style=3D"fon=
t-family:comic sans ms,sans-serif;color:rgb(11,83,148)">{code}<br><br></div=
><div class=3D"gmail_default" style=3D"font-family:comic sans ms,sans-serif=
;color:rgb(11,83,148)">The exception is:<br><br><pre style=3D"margin:0px;bo=
rder:0px;color:rgb(34,34,34);font-style:normal;font-variant:normal;font-wei=
ght:normal;letter-spacing:normal;line-height:normal;text-align:start;text-i=
ndent:0px;text-transform:none;word-spacing:0px;background-color:rgb(255,255=
,255)">WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running ch=
ild : java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cas=
t to org.apache.avro.generic.IndexedRecord
	at org.apache.crunch.types.avro.AvroTableType$IndexedRecordToPair.map(Avro=
TableType.java:88)
	at org.apache.crunch.MapFn.process(MapFn.java:34)
	at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98)
	at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:109)
	at org.apache.crunch.impl.mr.run.CrunchMapper.map(CrunchMapper.java:60)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati=
on.java:1671)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)</pre><br>Do=
es someone have an idea why the exception might be occurring? The version o=
f avro is 1.7.6. I have checked for dependency conflicts but there doesn=
9;t seem to be a problem with it although it looks like it.<br><br></div><d=
iv class=3D"gmail_default" style=3D"font-family:comic sans ms,sans-serif;co=
lor:rgb(11,83,148)">Thanks.<br></div></div>
</blockquote></div>

--001a113eca102c3046051ca67c4d--