Return-Path: X-Original-To: apmail-crunch-user-archive@www.apache.org Delivered-To: apmail-crunch-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id ABA96187C2 for ; Wed, 22 Jul 2015 18:00:58 +0000 (UTC) Received: (qmail 1300 invoked by uid 500); 22 Jul 2015 18:00:42 -0000 Delivered-To: apmail-crunch-user-archive@crunch.apache.org Received: (qmail 1260 invoked by uid 500); 22 Jul 2015 18:00:42 -0000 Mailing-List: contact user-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@crunch.apache.org Delivered-To: mailing list user@crunch.apache.org Received: (qmail 1249 invoked by uid 99); 22 Jul 2015 18:00:42 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2015 18:00:42 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 4F376D6BC3 for ; Wed, 22 Jul 2015 18:00:42 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.001 X-Spam-Level: *** X-Spam-Status: No, score=3.001 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id IaxGxK3PEj-v for ; Wed, 22 Jul 2015 18:00:31 +0000 (UTC) Received: from mail-yk0-f178.google.com (mail-yk0-f178.google.com [209.85.160.178]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 9FBFB20E8F for ; Wed, 22 Jul 2015 18:00:30 +0000 (UTC) Received: by ykay190 with SMTP id y190so198973564yka.3 for ; Wed, 22 Jul 2015 10:58:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=a/tJ7HNqxxQ+K4VXIwRHIQVwWMR/Z2eRTggX3szZJNk=; b=iJ8N4jJwFyMZBNZjuLKUwCVWzC4IMZXyi8FXHgRcC2EKY3VBZr/fqVcQc/si7qF6XY ZKCXQcJYYxLM7X7VoD3InhCOBCjIwctZ3VUaCmDnSaG4MHe7zaJ8JUFSih+9LsH7ZjyC QbatqSaliNSl3Q5UMZN2Wa+kEobzwWbi6Hcg/BxYEsjMF/3doQNOx9SFgJKGNQ4aZUB5 W1r31KdNek6CS4UKAR4ep5aF4OnAo/VRt7CM7ybmxEVLvRaLTnMpQYcmj6FjiTcH3zJG LE0tw18u7Y/azAPfx1kh9G4DZV/TK/m9i/GjlYm/wyUpothPmYop5YL3pXZyEgALb/dO myKg== X-Gm-Message-State: ALoCoQnMMmfsi4iCin3TYLhRC/BMrXq1YfkefKyWJxxVzTZ+5cXBdOB8xF3CxjEEa5O2iOhQsabe X-Received: by 10.170.229.133 with SMTP id v127mr3730729ykf.97.1437587934080; Wed, 22 Jul 2015 10:58:54 -0700 (PDT) MIME-Version: 1.0 Received: by 10.13.196.197 with HTTP; Wed, 22 Jul 2015 10:58:34 -0700 (PDT) In-Reply-To: References: From: Josh Wills Date: Wed, 22 Jul 2015 10:58:34 -0700 Message-ID: Subject: Re: ClassNotFoundException running with Oozie To: user@crunch.apache.org Content-Type: multipart/alternative; boundary=001a113bbab8252ece051b7a867a --001a113bbab8252ece051b7a867a Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Mike Baretta posted about a similar issue late last year and had an ugly fix that involved copying the Crunch jars into the distributed cache. You can see the whole thread here: https://www.mail-archive.com/user@crunch.apache.org/msg00438.html I myself haven't run into this one. J On Tue, Jul 21, 2015 at 2:12 PM, David Ortiz wrote: > Hello everyone, > > > > I=E2=80=99m getting an interesting exception running a crunch pipeli= ne from > Oozie. I have all the crunch dependencies bundled in a fat jar of > dependencies called crunch-lib. My avro schemas all live in a jar called > schemas. These all live in a sharelib directory for java actions on HDFS= . > My job itself is in a jar which lives in a directory pointed to by > oozie.libpath. As far as I can tell the Oozie job is getting all of the > dependencies since my crunch client code runs and tries to spin up MR > jobs. However, it fails, with the jobs it creates having the following > exception: > > > > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > java.lang.RuntimeException: java.lang.ClassNotFoundException: Class > org.apache.crunch.impl.mr.run.CrunchOutputFormat not found > > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:47= 2) > > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:45= 2) > > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRA= ppMaster.java:1541) > > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAp= pMaster.java:452) > > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.ja= va:371) > > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:149= 9) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation= .java:1671) > > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAp= pMaster.java:1496) > > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1429= ) > > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: > Class org.apache.crunch.impl.mr.run.CrunchOutputFormat not found > > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2112) > > at > org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobC= ontextImpl.java:232) > > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:46= 8) > > ... 11 more > > Caused by: java.lang.ClassNotFoundException: Class > org.apache.crunch.impl.mr.run.CrunchOutputFormat not found > > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:20= 18) > > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2110) > > ... 13 more > > > > > > Anyone have any ideas how the dependencies would be making it to the > crunch client, but not into the jar that crunch submits to the cluster? > > > > Thanks, > > Dave > *This email is intended only for the use of the individual(s) to whom it > is addressed. If you have received this communication in error, please > immediately notify the sender and delete the original email.* > --=20 Director of Data Science Cloudera Twitter: @josh_wills --001a113bbab8252ece051b7a867a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Mike Baretta posted about a similar issue late last year a= nd had an ugly fix that involved copying the Crunch jars into the distribut= ed cache. You can see the whole thread here:


I myself haven't run into this one.

J

On Tue, Jul 21, 2015 at 2:12 PM, David Ortiz <dortiz@videologygr= oup.com> wrote:

Hello everyone,

=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0 I=E2=80=99m getting an inte= resting exception running a crunch pipeline from Oozie.=C2=A0 I have all th= e crunch dependencies bundled in a fat jar of dependencies called crunch-li= b.=C2=A0 My avro schemas all live in a jar called schemas.=C2=A0 These all = live in a sharelib directory for java actions on HDFS.=C2=A0 My job itself is i= n a jar which lives in a directory pointed to by oozie.libpath.=C2=A0 As fa= r as I can tell the Oozie job is getting all of the dependencies since my c= runch client code runs and tries to spin up MR jobs.=C2=A0 However, it fails, with the jobs it creates having the foll= owing exception:

=C2=A0

org.apache.hadoop= .yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lan= g.ClassNotFoundException: Class org.apache.crunch.impl.mr.run.CrunchOutputF= ormat not found

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.v2.app.MRAppMast= er$1.call(MRAppMaster.java:472)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.v2.app.MRAppMast= er$1.call(MRAppMaster.java:452)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.v2.app.MRAppMast= er.callWithJobClassLoader(MRAppMaster.java:1541)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.v2.app.MRAppMast= er.createOutputCommitter(MRAppMaster.java:452)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.v2.app.MRAppMast= er.serviceInit(MRAppMaster.java:371)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.service.AbstractService.in= it(AbstractService.java:163)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.v2.app.MRAppMast= er$4.run(MRAppMaster.java:1499)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.security.AccessController.doPrivileged(= Native Method)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at javax.security.auth.Subject.doAs(Subject.jav= a:415)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.security.UserGroupInformat= ion.doAs(UserGroupInformation.java:1671)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.v2.app.MRAppMast= er.initAndStartAppMaster(MRAppMaster.java:1496)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.v2.app.MRAppMast= er.main(MRAppMaster.java:1429)

Caused by: java.l= ang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.cr= unch.impl.mr.run.CrunchOutputFormat not found

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.conf.Configuration.getClas= s(Configuration.java:2112)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.task.JobContextI= mpl.getOutputFormatClass(JobContextImpl.java:232)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.mapreduce.v2.app.MRAppMast= er$1.call(MRAppMaster.java:468)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 ... 11 more

Caused by: java.l= ang.ClassNotFoundException: Class org.apache.crunch.impl.mr.run.CrunchOutpu= tFormat not found

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.conf.Configuration.getClas= sByName(Configuration.java:2018)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.conf.Configuration.getClas= s(Configuration.java:2110)

=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 ... 13 more

=C2=A0

=C2=A0

Anyone have any ideas how the dependencies would be = making it to the crunch client, but not into the jar that crunch submits to= the cluster?

=C2=A0

Thanks,

=C2=A0=C2=A0=C2=A0 Dave

This email is intended only for the use of the individual(s) to whom it = is addressed. If you have received this communication in error, please imme= diately notify the sender and delete the original email.



--
Director of Data Science
Twitter: @josh_wills
--001a113bbab8252ece051b7a867a--