Return-Path: X-Original-To: apmail-crunch-user-archive@www.apache.org Delivered-To: apmail-crunch-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 52B171086F for ; Wed, 26 Nov 2014 17:31:54 +0000 (UTC) Received: (qmail 53896 invoked by uid 500); 26 Nov 2014 17:31:54 -0000 Delivered-To: apmail-crunch-user-archive@crunch.apache.org Received: (qmail 53861 invoked by uid 500); 26 Nov 2014 17:31:54 -0000 Mailing-List: contact user-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@crunch.apache.org Delivered-To: mailing list user@crunch.apache.org Received: (qmail 53842 invoked by uid 99); 26 Nov 2014 17:31:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Nov 2014 17:31:54 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mike.barretta@gmail.com designates 209.85.213.169 as permitted sender) Received: from [209.85.213.169] (HELO mail-ig0-f169.google.com) (209.85.213.169) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Nov 2014 17:31:28 +0000 Received: by mail-ig0-f169.google.com with SMTP id hl2so8874654igb.2 for ; Wed, 26 Nov 2014 09:29:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=T2BpjE6s+o+F8FEPNUajO7T0eaAh93QUyrGJHYk/ct0=; b=0XQUP9J8N6SlGn5tsMAUgh0Rx7wgMvWYbmTF2KjAtqssEfbzhMWqIsCK/ktuOgI5QV IQzNriC6MYBxC3R226bUG0hYGo/5XpLulATDg/8sXNvO4HOd8oBmCK7FDnCLi4kUk1yE wgUugWRhnyitdh77NSoEU4HVGwmkRwjlKSeQuDj1NqHJcxwoPcm5epuQRalXnPKMBalp wmAcKl5qNKDlVUc7ZaovbV0HafkY318f14nf/5SJB1tFAAm5CU0E1RlEaJabdQxqPIwg mo1XiN/Wo4T8nSeUCJq6VUxIpz5I00o3QcaqW4zpkt6a/Y40/Y37Vczm5E35ByZAGe9T Alcw== MIME-Version: 1.0 X-Received: by 10.42.176.66 with SMTP id bd2mr24098916icb.84.1417022997315; Wed, 26 Nov 2014 09:29:57 -0800 (PST) Received: by 10.64.117.164 with HTTP; Wed, 26 Nov 2014 09:29:57 -0800 (PST) In-Reply-To: References: Date: Wed, 26 Nov 2014 12:29:57 -0500 Message-ID: Subject: Re: ClassNotFoundException: Class org.apache.crunch.impl.mr.run.CrunchMapper From: Mike Barretta To: user@crunch.apache.org Content-Type: multipart/alternative; boundary=90e6ba6e89106514020508c66017 X-Virus-Checked: Checked by ClamAV on apache.org --90e6ba6e89106514020508c66017 Content-Type: text/plain; charset=UTF-8 Thank you for the quick reply. I am indeed using the Oozie workflow lib directory as described here: http://oozie.apache.org/docs/3.3.2/WorkflowFunctionalSpec.html#a7_Workflow_Application_Deployment. The primary job, which implements Tool, is able to run, it's just the jobs launched by the doFn() which fail. Is there a step where I might need to tell the Crunch pipeline about the jars loaded by Oozie? On Fri, Nov 21, 2014 at 5:27 PM, Micah Whitacre wrote: > The support of a lib folder inside of a jar is not necessarily guaranteed > to be supported on all versions of Hadoop.[1] > > We typically go with the "uber" jar where we use maven-shade-plugin to > actually explode the crunch dependencies and others into the assembly jar. > Another approach since you are using Oozie is to include the jar in the > workflow lib directory. That should put the jar on the classpath. The > last approach is obviously to manually use DistributedCache yourself which > will distribute it out to the cluster. > > [1] - > http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/ > > On Fri, Nov 21, 2014 at 4:15 PM, Mike Barretta > wrote: > >> All, >> >> I'm running an MRPipeline from crunch-core 0.11.0-hadoop2 on a CDH5.1 >> cluster via oozie. While the main job runs okay, the doFn() it calls fails >> due to the CNFE. The jar containing my classes does indeed contain >> lib/crunch-core-0.11.0-hadoop2.jar. >> >> Does the crunch jar need to be added to the hadoop lib on all nodes? It >> seems like that would/should be unnecessary. >> >> Thanks, >> Mike >> > > --90e6ba6e89106514020508c66017 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thank you for the quick reply.

I am indeed using the Oozie= workflow lib directory as described here: http://oozie.apache.org/docs/3.3.2/WorkflowFunctionalSpec.html#a7_Workfl= ow_Application_Deployment. =C2=A0

The primary job, which implement= s Tool, is able to run, it's just the jobs launched by the doFn() which= fail.=C2=A0 Is there a step where I might need to tell the Crunch pipeline= about the jars loaded by Oozie?

=
On Fri, Nov 21, 2014 at 5:27 PM, Micah Whitacre = <mkwhitacre@gmail.com> wrote:
The support of a lib folder inside of a jar is not = necessarily guaranteed to be supported on all versions of Hadoop.[1]=C2=A0<= div>
We typically go with the "uber" jar where we u= se maven-shade-plugin to actually explode the crunch dependencies and other= s into the assembly jar.=C2=A0 Another approach since you are using Oozie i= s to include the jar in the workflow lib directory.=C2=A0 That should put t= he jar on the classpath.=C2=A0 The last approach is obviously to manually u= se DistributedCache yourself which will distribute it out to the cluster.
<= div class=3D"h5">

= On Fri, Nov 21, 2014 at 4:15 PM, Mike Barretta <mike.barretta@gmail.= com> wrote:
All= ,

I'm running an MRPipeline from crunch-core 0.11.0-hadoop2=C2=A0o= n a CDH5.1 cluster via oozie.=C2=A0 While the main job runs okay, the doFn(= ) it calls fails due to the CNFE.=C2=A0 The jar containing my classes does = indeed contain lib/crunch-core-0.11.0-hadoop2.jar.

Does the crunch jar= need to be added to the hadoop lib on all nodes?=C2=A0 It seems like that = would/should be unnecessary.

Thanks,
Mike=C2=A0


--90e6ba6e89106514020508c66017--