Return-Path: Delivered-To: apmail-hadoop-mapreduce-dev-archive@minotaur.apache.org Received: (qmail 13977 invoked from network); 5 Feb 2011 14:18:31 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 5 Feb 2011 14:18:31 -0000 Received: (qmail 83056 invoked by uid 500); 5 Feb 2011 14:18:30 -0000 Delivered-To: apmail-hadoop-mapreduce-dev-archive@hadoop.apache.org Received: (qmail 82663 invoked by uid 500); 5 Feb 2011 14:18:27 -0000 Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-dev@hadoop.apache.org Delivered-To: mailing list mapreduce-dev@hadoop.apache.org Received: (qmail 82649 invoked by uid 99); 5 Feb 2011 14:18:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Feb 2011 14:18:26 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.161.48 as permitted sender) Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Feb 2011 14:18:19 +0000 Received: by fxm2 with SMTP id 2so3474582fxm.35 for ; Sat, 05 Feb 2011 06:17:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=Tefkg7iE5zBHRrUu3gybLG6TBhuDuF3HFMikoJHSfgY=; b=X+OY6JFjkqgjFpFQlgqD4OcWU7H0BZYQB/RpocBY4zmK4+xq4s1cWy548UvIAG6Asc f5shIF2WWUq1uMayeCcE2dghQSAagDdEvrNFB6IYDYCu9N7aD9iREFhrMal1T7/VE8oS HL8KdcwJ4a9/DmzQCK6d7BYTi2FFhd4St416U= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=f25OKXoGGmZ30f2/3iNKss8fk3pW07tAdejS1xoVsS9xQD1pDiarvbAJ4obseqY+N4 befV8n2951XbXVZNSfwa+qJalRWbARmCYr/NE6jy+9UIOZWqcj0v3yEW6G8OgUNLJ2ML /kEVFMhiW1DYl8hHc7JPC4qfy3MTVlcJo8tHE= MIME-Version: 1.0 Received: by 10.223.70.193 with SMTP id e1mr1343849faj.91.1296915478513; Sat, 05 Feb 2011 06:17:58 -0800 (PST) Received: by 10.223.78.140 with HTTP; Sat, 5 Feb 2011 06:17:58 -0800 (PST) Date: Sat, 5 Feb 2011 06:17:58 -0800 Message-ID: Subject: DistributedCache.addFileToClassPath() From: Ted Yu To: mapreduce-dev@hadoop.apache.org, cdh-dev@cloudera.org Content-Type: multipart/alternative; boundary=20cf3054a2b3668758049b89aa7e --20cf3054a2b3668758049b89aa7e Content-Type: text/plain; charset=ISO-8859-1 Hi, We use cdh3b2. Recently we experience map task failure because of: INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - 2011-02-05 02:17:23,855 WARN org.apache.hadoop.mapred.TaskTracker: Error running child INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - java.io.IOException: Split class com.carrieriq.m2m.platform.mmp2.input.FileListInputSplit not found INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:326) INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - at org.apache.hadoop.mapred.Child.main(Child.java:170) INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - Caused by: java.lang.ClassNotFoundException: com.carrieriq.m2m.platform.mmp2.input.FileListInputSplit INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - at java.net.URLClassLoader$1.run(URLClassLoader.java:202) INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - at java.security.AccessController.doPrivileged(Native Method) INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - at java.net.URLClassLoader.findClass(URLClassLoader.java:190) INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - at java.lang.ClassLoader.loadClass(ClassLoader.java:307) INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - at java.lang.ClassLoader.loadClass(ClassLoader.java:248) INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - at java.lang.Class.forName0(Native Method) INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - at java.lang.Class.forName(Class.java:247) INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:907) INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:323) INFO [2011-02-04 18:17:36] (ExecUtil.java:261) - ... 2 more I found that the following config parameter was missing from the underlying Job Conf: mapred.job.classpath.files We use the following code: Path dest = copyToDfs(jar, jobConf); // add URL into class path for grid based job also to ensure flow validation can work using mmp command run ClassUtil.addURL(jar.toURL()); DistributedCache.addFileToClassPath(dest, jobConf); >From log, I verified that ClassUtil.addURL() was called. The following API doesn't have return code, no logging: public static void addFileToClassPath(Path file, Configuration conf) Please advise how I can debug this issue. Thanks --20cf3054a2b3668758049b89aa7e--