crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Beech <d...@paraliatech.com>
Subject Re: Issue with Crunch + avros on EMR 1.0.3
Date Tue, 22 Oct 2013 08:09:45 GMT
Hi Som

As well as the avro jars as Josh said, paranamer needs to be included in
the fat jar too.

Cheers
Dave

On Tuesday, October 22, 2013, Som Satpathy wrote:

> Hi Josh,
>
> Thanks for the pointer. Yes, the job fat jar is complied against 2.0.0-mr1-cdh4.2.0.
> I was under the assumption that 2.0.0-mr1-cdh4.2.0 is based on hadoop 1.x
> APIs. I will try again, after having it compiled against 1.0.3 hadoop-core.
>
> The job fat jar includes avro-1.7.4 and avro-mapred. So, I found it weird
> as to why EMR was complaining about it.
>
> Thanks,
> Som
>
>
> On Mon, Oct 21, 2013 at 8:50 PM, Josh Wills <jwills@cloudera.com<javascript:_e({},
'cvml', 'jwills@cloudera.com');>
> > wrote:
>
>> Hey Som,
>>
>> I didn't quite parse this; you're running a Crunch EMR job (which has an
>> API based on Hadoop 1.0.3), but compiling the jar against
>> 2.0.0-mr1-cdh4.2.0? I don't see how that would get very far b/c the APIs
>> are different; you should compile your local job against Hadoop 1.0.3.
>>
>> From the look of the error, I suspect that the Avro jars (avro-1.7.4.jar
>> and avro-mapred-1.7.4.jar) aren't included in the -libjars for the job
>> submission.
>>
>> J
>>
>>
>> On Mon, Oct 21, 2013 at 7:57 PM, Som Satpathy <somsatpathy@gmail.com<javascript:_e({},
'cvml', 'somsatpathy@gmail.com');>
>> > wrote:
>>
>>> Hi all,
>>>
>>> I am trying to run a crunch job that does a parallelDo who's return
>>> PType is a Avros.specifics() on EMR 1.0.3. But job always throws the below
>>> error:
>>>
>>>
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> com/thoughtworks/paranamer/Paranamer
>>>         at
>>> org.apache.avro.reflect.ReflectDatumReader.<init>(ReflectDatumReader.java:49)
>>>         at
>>> org.apache.crunch.types.avro.AvroCapabilities.canDecodeSpecificSchemaWithReflectDatumReader(AvroCapabilities.java:81)
>>>         at org.apache.crunch.types.avro.Avros.<clinit>(Avros.java:84)
>>>         at com.research.parser.Explorer.run(Explorer.java:110)
>>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>         at com.research.parser. Explorer.main(Explorer.java:82)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>         at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>         at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
>>> Caused by: java.lang.ClassNotFoundException:
>>> com.thoughtworks.paranamer.Paranamer
>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>         ... 11 more
>>>
>>> I made sure that my project is using hadoop version 2.0.0-mr1-cdh4.2.0,
>>> crunch version 0.7.0, and avro 1.7.4. However, i'm still getting the above
>>> error while executing my job.
>>>
>>> I would really appreciate any pointers to help fix this problem.
>>>
>>> Thanks,
>>> Som
>>>
>>
>>
>>
>> --
>> Director of Data Science
>> Cloudera <http://www.cloudera.com>
>> Twitter: @josh_wills <http://twitter.com/josh_wills>
>>
>
>

Mime
View raw message