Return-Path: X-Original-To: apmail-crunch-user-archive@www.apache.org Delivered-To: apmail-crunch-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D7B3018069 for ; Tue, 21 Jul 2015 21:12:26 +0000 (UTC) Received: (qmail 74636 invoked by uid 500); 21 Jul 2015 21:12:14 -0000 Delivered-To: apmail-crunch-user-archive@crunch.apache.org Received: (qmail 74598 invoked by uid 500); 21 Jul 2015 21:12:14 -0000 Mailing-List: contact user-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@crunch.apache.org Delivered-To: mailing list user@crunch.apache.org Received: (qmail 74587 invoked by uid 99); 21 Jul 2015 21:12:14 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2015 21:12:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id A338518B066 for ; Tue, 21 Jul 2015 21:12:13 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.89 X-Spam-Level: * X-Spam-Status: No, score=1.89 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-1.108, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Igcs9Ymm6Kle for ; Tue, 21 Jul 2015 21:12:11 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2on0074.outbound.protection.outlook.com [65.55.169.74]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 256B843DF3 for ; Tue, 21 Jul 2015 21:12:11 +0000 (UTC) Received: from BY2PR07MB859.namprd07.prod.outlook.com (10.242.233.26) by BY2PR07MB1507.namprd07.prod.outlook.com (10.162.77.14) with Microsoft SMTP Server (TLS) id 15.1.213.14; Tue, 21 Jul 2015 21:12:05 +0000 Received: from BY2PR07MB858.namprd07.prod.outlook.com (10.242.233.22) by BY2PR07MB859.namprd07.prod.outlook.com (10.242.233.26) with Microsoft SMTP Server (TLS) id 15.1.213.14; Tue, 21 Jul 2015 21:12:03 +0000 Received: from BY2PR07MB858.namprd07.prod.outlook.com ([10.242.233.22]) by BY2PR07MB858.namprd07.prod.outlook.com ([10.242.233.22]) with mapi id 15.01.0213.021; Tue, 21 Jul 2015 21:12:03 +0000 From: David Ortiz To: "user@crunch.apache.org" Subject: ClassNotFoundException running with Oozie Thread-Topic: ClassNotFoundException running with Oozie Thread-Index: AdDD+bVB8qyk6MohQ4uEtsWnn4PILQ== Date: Tue, 21 Jul 2015 21:12:02 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: crunch.apache.org; dkim=none (message not signed) header.d=none; x-originating-ip: [209.124.171.65] x-microsoft-exchange-diagnostics: 1;BY2PR07MB859;5:QNngduBOYjczingtucLZQm4k5rDA14obqH8tt83rrj7xtM6I7SG05PWZR50SdoYeuEiTUDj4Bj5Z2CAbWoCZWGQ0hRMGtjzn0TsQNjXLQb0xHzobXGS5iZ+EPbYwhv3Mo8I0uHrjbSqQHWRfwXCY4Q==;24:Wcb/hl6x7DkGT87Gv5fZZrgO20KN9wDLynS/7x6EVbHFEtweZElwcSGbwdPopkK1IJG/qiIGN9BQ1QlT/PNevGt1QHWjY42L04SKZeaWz1I=;20:qBVVmwSMXw80+GqQ4+aAbh6LqqKqAIyirDg49GEAEmxsmszvYCtLVW4fHOl0ayROooCjktbl3BdcM5vVzdi40g== x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY2PR07MB859;UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY2PR07MB1507; by2pr07mb859: X-MS-Exchange-Organization-RulesExecuted x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(108003899814671); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(601004)(5005006)(3002001);SRVR:BY2PR07MB859;BCL:0;PCL:0;RULEID:;SRVR:BY2PR07MB859; x-forefront-prvs: 0644578634 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(6009001)(164054003)(53754006)(19625215002)(33656002)(54356999)(16236675004)(99286002)(86362001)(50986999)(66066001)(74316001)(229853001)(2351001)(19580395003)(5003600100002)(2656002)(2501003)(76576001)(40100003)(450100001)(5001960100002)(110136002)(19300405004)(77156002)(62966003)(2900100001)(107886002)(92566002)(189998001)(77096005)(46102003)(15975445007)(102836002)(87936001)(5002640100001);DIR:OUT;SFP:1101;SCL:1;SRVR:BY2PR07MB859;H:BY2PR07MB858.namprd07.prod.outlook.com;FPR:;SPF:None;MLV:sfv;LANG:en; Content-Type: multipart/alternative; boundary="_000_BY2PR07MB858FE4875DB91EFD5CD7B90B8840BY2PR07MB858namprd_" MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Jul 2015 21:12:02.8359 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 85564c82-a058-4b99-8b37-19832d4dc76a X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR07MB859 X-Microsoft-Exchange-Diagnostics: 1;BY2PR07MB1507;2:uf5ySd6YzOih+JTFoVF4Q5XhsUaZJy4CE1O/enKYz/KqQy8TbCvjB9+MZOoyynbV;3:7WF3/M3szMlhnxIsMVrHIrWhxOirUYr6SFsVuWbEOG/jZAIVlCJoHKbls2eG/vqHbO/TKcUkAP7ht/WpHwTFbEgBT/SbiRjbaejiwT/u8hE4VCmxfQaJcSkPLFIAih0Ygus3rfyo25/gasUt4NQYKQ==;25:Vf4o7KnMQZ5HxGuiGQdD3NtPKNQLHvd/O2LzwC3emNC/z7jbhADpXhdD7jYIJ3dsGN58CUGiycvgBOU+UR173LdGvMq3ihcPv9leub5aTrgBGXFKFBdOW0bovq3d1THndhx65dOI4bVEKEHfByIZeToN/2Wl+GeBDaBMyOH28f7oOGAAYxSvzCLiWFT0EIEX5bvls4eOcQVJKWsNx5ub4/R0moTdLSEo7vIU3WcEGqJb+l2tw8HUYOOotdj4LxqhuH9pK9xLX3wH4ozTR533Gg==;20:GnJ4E2DcRcM8ixWuNoBTRWOtGHkLhVIbu28LPmBlqK59taHWnY/0HiRwlYM6VM2t+cEYzfqZmZaeQrTxrNAPFg==;23:WM4SsKtVnwYOKQxVhetQ4IOtdexrnSv4S1SfVJOqaa6CmrXJqTSTdwTU1LKn3MxVPNPXtBQRRROivJJexpPtpSJ6AxiGUIHz4WwSMXL81GXP/VWYCNY/h1b8JvrUs8CKiRqTIs5I26XbLi791btduW2NFP6Ct8YQho1zSS4pqJfFiYDcCEfKMZf07ws/5lRf+KsG1gEOjNzQp7Lpsk4JSvY+x8r9+iGzqodX72tzISF3duxwl6ev1V5Ghf3sEe5K BY2PR07MB1507: X-MS-Exchange-Organization-RulesExecuted X-OriginatorOrg: videologygroup.com --_000_BY2PR07MB858FE4875DB91EFD5CD7B90B8840BY2PR07MB858namprd_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello everyone, I'm getting an interesting exception running a crunch pipeline from Oo= zie. I have all the crunch dependencies bundled in a fat jar of dependenci= es called crunch-lib. My avro schemas all live in a jar called schemas. T= hese all live in a sharelib directory for java actions on HDFS. My job its= elf is in a jar which lives in a directory pointed to by oozie.libpath. As= far as I can tell the Oozie job is getting all of the dependencies since m= y crunch client code runs and tries to spin up MR jobs. However, it fails,= with the jobs it creates having the following exception: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeEx= ception: java.lang.ClassNotFoundException: Class org.apache.crunch.impl.mr.= run.CrunchOutputFormat not found at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaste= r.java:472) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaste= r.java:452) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassL= oader(MRAppMaster.java:1541) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommi= tter(MRAppMaster.java:452) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRApp= Master.java:371) at org.apache.hadoop.service.AbstractService.init(AbstractService.j= ava:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster= .java:1499) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupIn= formation.java:1671) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMa= ster(MRAppMaster.java:1496) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.= java:1429) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Cl= ass org.apache.crunch.impl.mr.run.CrunchOutputFormat not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java= :2112) at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatC= lass(JobContextImpl.java:232) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaste= r.java:468) ... 11 more Caused by: java.lang.ClassNotFoundException: Class org.apache.crunch.impl.m= r.run.CrunchOutputFormat not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuratio= n.java:2018) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java= :2110) ... 13 more Anyone have any ideas how the dependencies would be making it to the crunch= client, but not into the jar that crunch submits to the cluster? Thanks, Dave This email is intended only for the use of the individual(s) to whom it is = addressed. If you have received this communication in error, please immedia= tely notify the sender and delete the original email. --_000_BY2PR07MB858FE4875DB91EFD5CD7B90B8840BY2PR07MB858namprd_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello everyone,

 

     I’m getting an intere= sting exception running a crunch pipeline from Oozie.  I have all the = crunch dependencies bundled in a fat jar of dependencies called crunch-lib.=   My avro schemas all live in a jar called schemas.  These all li= ve in a sharelib directory for java actions on HDFS.  My job itself is i= n a jar which lives in a directory pointed to by oozie.libpath.  As fa= r as I can tell the Oozie job is getting all of the dependencies since my c= runch client code runs and tries to spin up MR jobs.  However, it fails, with the jobs it creates having the foll= owing exception:

 

org.apache.hadoop= .yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lan= g.ClassNotFoundException: Class org.apache.crunch.impl.mr.run.CrunchOutputF= ormat not found

   = ;     at org.apache.hadoop.mapreduce.v2.app.MRAppMaster= $1.call(MRAppMaster.java:472)

   = ;     at org.apache.hadoop.mapreduce.v2.app.MRAppMaster= $1.call(MRAppMaster.java:452)

   = ;     at org.apache.hadoop.mapreduce.v2.app.MRAppMaster= .callWithJobClassLoader(MRAppMaster.java:1541)

   = ;     at org.apache.hadoop.mapreduce.v2.app.MRAppMaster= .createOutputCommitter(MRAppMaster.java:452)

   = ;     at org.apache.hadoop.mapreduce.v2.app.MRAppMaster= .serviceInit(MRAppMaster.java:371)

   = ;     at org.apache.hadoop.service.AbstractService.init= (AbstractService.java:163)

   = ;     at org.apache.hadoop.mapreduce.v2.app.MRAppMaster= $4.run(MRAppMaster.java:1499)

   = ;     at java.security.AccessController.doPrivileged(Na= tive Method)

   = ;     at javax.security.auth.Subject.doAs(Subject.java:= 415)

   = ;     at org.apache.hadoop.security.UserGroupInformatio= n.doAs(UserGroupInformation.java:1671)

   = ;     at org.apache.hadoop.mapreduce.v2.app.MRAppMaster= .initAndStartAppMaster(MRAppMaster.java:1496)

   = ;     at org.apache.hadoop.mapreduce.v2.app.MRAppMaster= .main(MRAppMaster.java:1429)

Caused by: java.l= ang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.cr= unch.impl.mr.run.CrunchOutputFormat not found

   = ;     at org.apache.hadoop.conf.Configuration.getClass(= Configuration.java:2112)

   = ;     at org.apache.hadoop.mapreduce.task.JobContextImp= l.getOutputFormatClass(JobContextImpl.java:232)

   = ;     at org.apache.hadoop.mapreduce.v2.app.MRAppMaster= $1.call(MRAppMaster.java:468)

   = ;     ... 11 more

Caused by: java.l= ang.ClassNotFoundException: Class org.apache.crunch.impl.mr.run.CrunchOutpu= tFormat not found

   = ;     at org.apache.hadoop.conf.Configuration.getClassB= yName(Configuration.java:2018)

   = ;     at org.apache.hadoop.conf.Configuration.getClass(= Configuration.java:2110)

   = ;     ... 13 more

 

 

Anyone have any ideas how the dependencies would be = making it to the crunch client, but not into the jar that crunch submits to= the cluster?

 

Thanks,

    Dave

This email is intended only for the use of the individual(s) to whom it = is addressed. If you have received this communication in error, please imme= diately notify the sender and delete the original email. --_000_BY2PR07MB858FE4875DB91EFD5CD7B90B8840BY2PR07MB858namprd_--