Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
Received-SPF: pass (athena.apache.org: domain of mkwhitacre@gmail.com
 designates 209.85.214.182 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CA+1QwBiOjUA=52zSOpEosx_NqpAesgNfnikcxR2VupqTUrJsuA@mail.gmail.com>
References: 
 <CA+1QwBiOjUA=52zSOpEosx_NqpAesgNfnikcxR2VupqTUrJsuA@mail.gmail.com>
Date: Fri, 21 Nov 2014 16:27:33 -0600
Message-ID: 
 <CADH-_NLKf45LXaFdsGs11+wXCcXiES8FZt3kp8kDMfpLyEs+=Q@mail.gmail.com>
Subject: Re: ClassNotFoundException: Class
 org.apache.crunch.impl.mr.run.CrunchMapper
From: Micah Whitacre <mkwhitacre@gmail.com>
To: user@crunch.apache.org
Content-Type: multipart/alternative; boundary=001a11403e7a7e47f7050865f3ef

--001a11403e7a7e47f7050865f3ef
Content-Type: text/plain; charset=UTF-8

The support of a lib folder inside of a jar is not necessarily guaranteed
to be supported on all versions of Hadoop.[1]

We typically go with the "uber" jar where we use maven-shade-plugin to
actually explode the crunch dependencies and others into the assembly jar.
Another approach since you are using Oozie is to include the jar in the
workflow lib directory.  That should put the jar on the classpath.  The
last approach is obviously to manually use DistributedCache yourself which
will distribute it out to the cluster.

[1] -
http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/

On Fri, Nov 21, 2014 at 4:15 PM, Mike Barretta <mike.barretta@gmail.com>
wrote:

> All,
>
> I'm running an MRPipeline from crunch-core 0.11.0-hadoop2 on a CDH5.1
> cluster via oozie.  While the main job runs okay, the doFn() it calls fails
> due to the CNFE.  The jar containing my classes does indeed contain
> lib/crunch-core-0.11.0-hadoop2.jar.
>
> Does the crunch jar need to be added to the hadoop lib on all nodes?  It
> seems like that would/should be unnecessary.
>
> Thanks,
> Mike
>

--001a11403e7a7e47f7050865f3ef
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">The support of a lib folder inside of a jar is not necessa=
rily guaranteed to be supported on all versions of Hadoop.[1]=C2=A0<div><br=
></div><div>We typically go with the &quot;uber&quot; jar where we use mave=
n-shade-plugin to actually explode the crunch dependencies and others into =
the assembly jar.=C2=A0 Another approach since you are using Oozie is to in=
clude the jar in the workflow lib directory.=C2=A0 That should put the jar =
on the classpath.=C2=A0 The last approach is obviously to manually use Dist=
ributedCache yourself which will distribute it out to the cluster.<div><br>=
</div><div>[1] - <a href=3D"http://blog.cloudera.com/blog/2011/01/how-to-in=
clude-third-party-libraries-in-your-map-reduce-job/">http://blog.cloudera.c=
om/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job=
/</a><br></div></div></div><div class=3D"gmail_extra"><br><div class=3D"gma=
il_quote">On Fri, Nov 21, 2014 at 4:15 PM, Mike Barretta <span dir=3D"ltr">=
&lt;<a href=3D"mailto:mike.barretta@gmail.com" target=3D"_blank">mike.barre=
tta@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div =
dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:verdana,sans-=
serif">All,</div><div class=3D"gmail_default" style=3D"font-family:verdana,=
sans-serif"><br></div><div class=3D"gmail_default" style=3D"font-family:ver=
dana,sans-serif">I&#39;m running an MRPipeline from crunch-core 0.11.0-hado=
op2=C2=A0on a CDH5.1 cluster via oozie.=C2=A0 While the main job runs okay,=
 the doFn() it calls fails due to the CNFE.=C2=A0 The jar containing my cla=
sses does indeed contain lib/crunch-core-0.11.0-hadoop2.jar.</div><div clas=
s=3D"gmail_default" style=3D"font-family:verdana,sans-serif"><br></div><div=
 class=3D"gmail_default" style=3D"font-family:verdana,sans-serif">Does the =
crunch jar need to be added to the hadoop lib on all nodes?=C2=A0 It seems =
like that would/should be unnecessary.</div><div class=3D"gmail_default" st=
yle=3D"font-family:verdana,sans-serif"><br></div><div class=3D"gmail_defaul=
t" style=3D"font-family:verdana,sans-serif">Thanks,</div><div class=3D"gmai=
l_default" style=3D"font-family:verdana,sans-serif">Mike=C2=A0</div></div>
</blockquote></div><br></div>

--001a11403e7a7e47f7050865f3ef--