Return-Path: Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: (qmail 79557 invoked from network); 21 May 2010 05:35:39 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 21 May 2010 05:35:39 -0000 Received: (qmail 78376 invoked by uid 500); 21 May 2010 05:35:38 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 78286 invoked by uid 500); 21 May 2010 05:35:38 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 78278 invoked by uid 99); 21 May 2010 05:35:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 May 2010 05:35:38 +0000 X-ASF-Spam-Status: No, hits=-1450.9 required=10.0 tests=ALL_TRUSTED,AWL X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 May 2010 05:35:37 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o4L5ZG36008925 for ; Fri, 21 May 2010 05:35:17 GMT Message-ID: <14909988.12971274420116908.JavaMail.jira@thor> Date: Fri, 21 May 2010 01:35:16 -0400 (EDT) From: "Amareshwari Sriramadasu (JIRA)" To: mapreduce-issues@hadoop.apache.org Subject: [jira] Commented: (MAPREDUCE-1697) Document the behavior of -file option in streaming In-Reply-To: <15523467.15471271135090018.JavaMail.jira@thor> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869876#action_12869876 ] Amareshwari Sriramadasu commented on MAPREDUCE-1697: ---------------------------------------------------- Test failures are clearly not related to the patch. They failed because of NoClassDefFoundError (MAPREDUCE-1275) > Document the behavior of -file option in streaming > -------------------------------------------------- > > Key: MAPREDUCE-1697 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1697 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming, documentation > Affects Versions: 0.20.1 > Reporter: Amareshwari Sriramadasu > Fix For: 0.21.0, 0.22.0 > > Attachments: patch-1697.txt > > > The behavior of -file option in streaming is not documented anywhere. > The behavior of -file is the following : > 1) All the files passed through -file option are packaged into job.jar. > 2) If -file option is used for .class or .jar files, they are unjarred on tasktracker and placed in ${mapred.local.dir}/taskTracker/jobcache/job_ID/jars/classes or /lib, respectively. Symlinks to the directories classes and lib are created from the cwd of the task, . The names of symlinks are "classes", "lib". So file names of .class or .jar files do not appear in cwd of the task. > Paths to these files are automatically added to classpath. The tricky part is that hadoop framework can pick .class or .jar using classpath, but actual mapper script cannot. If you'd like to access these .class or .jar inside script, please do something like "java -cp lib/*;classes/* ". > 3) If -file option is used for files other than .class or .jar (e.g, .txt or .pl), these files are unjarred into ${mapred.local.dir}/taskTracker/jobcache/job_ID/jars/. Symlinks to these files are created from the cwd of the task. Names of these symlinks are actually file names. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.