crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Som Satpathy <somsatpa...@gmail.com>
Subject Re: Issue with Crunch + avros on EMR 1.0.3
Date Tue, 22 Oct 2013 22:15:30 GMT
Thanks for sharing the info Dave.

Som

On Tue, Oct 22, 2013 at 1:09 AM, Dave Beech <dave@paraliatech.com> wrote:

> Hi Som
>
> As well as the avro jars as Josh said, paranamer needs to be included in
> the fat jar too.
>
> Cheers
> Dave
>
>
> On Tuesday, October 22, 2013, Som Satpathy wrote:
>
>> Hi Josh,
>>
>> Thanks for the pointer. Yes, the job fat jar is complied against 2.0.0-mr1-cdh4.2.0.
>> I was under the assumption that 2.0.0-mr1-cdh4.2.0 is based on hadoop
>> 1.x APIs. I will try again, after having it compiled against 1.0.3
>> hadoop-core.
>>
>> The job fat jar includes avro-1.7.4 and avro-mapred. So, I found it weird
>> as to why EMR was complaining about it.
>>
>> Thanks,
>> Som
>>
>>
>> On Mon, Oct 21, 2013 at 8:50 PM, Josh Wills <jwills@cloudera.com> wrote:
>>
>>> Hey Som,
>>>
>>> I didn't quite parse this; you're running a Crunch EMR job (which has an
>>> API based on Hadoop 1.0.3), but compiling the jar against
>>> 2.0.0-mr1-cdh4.2.0? I don't see how that would get very far b/c the APIs
>>> are different; you should compile your local job against Hadoop 1.0.3.
>>>
>>> From the look of the error, I suspect that the Avro jars (avro-1.7.4.jar
>>> and avro-mapred-1.7.4.jar) aren't included in the -libjars for the job
>>> submission.
>>>
>>> J
>>>
>>>
>>> On Mon, Oct 21, 2013 at 7:57 PM, Som Satpathy <somsatpathy@gmail.com>wrote:
>>>
>>>> Hi all,
>>>>
>>>> I am trying to run a crunch job that does a parallelDo who's return
>>>> PType is a Avros.specifics() on EMR 1.0.3. But job always throws the below
>>>> error:
>>>>
>>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> com/thoughtworks/paranamer/Paranamer
>>>>         at
>>>> org.apache.avro.reflect.ReflectDatumReader.<init>(ReflectDatumReader.java:49)
>>>>         at
>>>> org.apache.crunch.types.avro.AvroCapabilities.canDecodeSpecificSchemaWithReflectDatumReader(AvroCapabilities.java:81)
>>>>         at org.apache.crunch.types.avro.Avros.<clinit>(Avros.java:84)
>>>>         at com.research.parser.Explorer.run(Explorer.java:110)
>>>>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>         at com.research.parser. Explorer.main(Explorer.java:82)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>         at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>         at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> com.thoughtworks.paranamer.Paranamer
>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>         ... 11 more
>>>>
>>>> I made sure that my project is using hadoop version 2.0.0-mr1-cdh4.2.0,
>>>> crunch version 0.7.0, and avro 1.7.4. However, i'm still getting the above
>>>> error while executing my job.
>>>>
>>>> I would really appreciate any pointers to help fix this problem.
>>>>
>>>> Thanks,
>>>> Som
>>>>
>>>
>>>
>>>
>>> --
>>> Director of Data Science
>>> Cloudera <http://www.cloudera.com>
>>> Twitter: @josh_wills <http://twitter.com/josh_wills>
>>>
>>
>>

Mime
View raw message