crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: Issue with Crunch + avros on EMR 1.0.3
Date Tue, 22 Oct 2013 03:50:41 GMT
Hey Som,

I didn't quite parse this; you're running a Crunch EMR job (which has an
API based on Hadoop 1.0.3), but compiling the jar against
2.0.0-mr1-cdh4.2.0? I don't see how that would get very far b/c the APIs
are different; you should compile your local job against Hadoop 1.0.3.

>From the look of the error, I suspect that the Avro jars (avro-1.7.4.jar
and avro-mapred-1.7.4.jar) aren't included in the -libjars for the job
submission.

J


On Mon, Oct 21, 2013 at 7:57 PM, Som Satpathy <somsatpathy@gmail.com> wrote:

> Hi all,
>
> I am trying to run a crunch job that does a parallelDo who's return PType
> is a Avros.specifics() on EMR 1.0.3. But job always throws the below error:
>
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/thoughtworks/paranamer/Paranamer
>         at
> org.apache.avro.reflect.ReflectDatumReader.<init>(ReflectDatumReader.java:49)
>         at
> org.apache.crunch.types.avro.AvroCapabilities.canDecodeSpecificSchemaWithReflectDatumReader(AvroCapabilities.java:81)
>         at org.apache.crunch.types.avro.Avros.<clinit>(Avros.java:84)
>         at com.research.parser.Explorer.run(Explorer.java:110)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at com.research.parser. Explorer.main(Explorer.java:82)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
> Caused by: java.lang.ClassNotFoundException:
> com.thoughtworks.paranamer.Paranamer
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         ... 11 more
>
> I made sure that my project is using hadoop version 2.0.0-mr1-cdh4.2.0,
> crunch version 0.7.0, and avro 1.7.4. However, i'm still getting the above
> error while executing my job.
>
> I would really appreciate any pointers to help fix this problem.
>
> Thanks,
> Som
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
View raw message