crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Som Satpathy <somsatpa...@gmail.com>
Subject Re: Issue with Crunch + avros on EMR 1.0.3
Date Tue, 22 Oct 2013 04:10:34 GMT
Hi Josh,

Thanks for the pointer. Yes, the job fat jar is complied against
2.0.0-mr1-cdh4.2.0.
I was under the assumption that 2.0.0-mr1-cdh4.2.0 is based on hadoop 1.x
APIs. I will try again, after having it compiled against 1.0.3 hadoop-core.

The job fat jar includes avro-1.7.4 and avro-mapred. So, I found it weird
as to why EMR was complaining about it.

Thanks,
Som


On Mon, Oct 21, 2013 at 8:50 PM, Josh Wills <jwills@cloudera.com> wrote:

> Hey Som,
>
> I didn't quite parse this; you're running a Crunch EMR job (which has an
> API based on Hadoop 1.0.3), but compiling the jar against
> 2.0.0-mr1-cdh4.2.0? I don't see how that would get very far b/c the APIs
> are different; you should compile your local job against Hadoop 1.0.3.
>
> From the look of the error, I suspect that the Avro jars (avro-1.7.4.jar
> and avro-mapred-1.7.4.jar) aren't included in the -libjars for the job
> submission.
>
> J
>
>
> On Mon, Oct 21, 2013 at 7:57 PM, Som Satpathy <somsatpathy@gmail.com>wrote:
>
>> Hi all,
>>
>> I am trying to run a crunch job that does a parallelDo who's return PType
>> is a Avros.specifics() on EMR 1.0.3. But job always throws the below error:
>>
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> com/thoughtworks/paranamer/Paranamer
>>         at
>> org.apache.avro.reflect.ReflectDatumReader.<init>(ReflectDatumReader.java:49)
>>         at
>> org.apache.crunch.types.avro.AvroCapabilities.canDecodeSpecificSchemaWithReflectDatumReader(AvroCapabilities.java:81)
>>         at org.apache.crunch.types.avro.Avros.<clinit>(Avros.java:84)
>>         at com.research.parser.Explorer.run(Explorer.java:110)
>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>         at com.research.parser. Explorer.main(Explorer.java:82)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
>> Caused by: java.lang.ClassNotFoundException:
>> com.thoughtworks.paranamer.Paranamer
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>         ... 11 more
>>
>> I made sure that my project is using hadoop version 2.0.0-mr1-cdh4.2.0,
>> crunch version 0.7.0, and avro 1.7.4. However, i'm still getting the above
>> error while executing my job.
>>
>> I would really appreciate any pointers to help fix this problem.
>>
>> Thanks,
>> Som
>>
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>

Mime
View raw message