Return-Path: Delivered-To: apmail-hadoop-core-commits-archive@www.apache.org Received: (qmail 92038 invoked from network); 17 Jul 2008 19:17:46 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 17 Jul 2008 19:17:46 -0000 Received: (qmail 93493 invoked by uid 500); 17 Jul 2008 19:17:45 -0000 Delivered-To: apmail-hadoop-core-commits-archive@hadoop.apache.org Received: (qmail 93462 invoked by uid 500); 17 Jul 2008 19:17:45 -0000 Mailing-List: contact core-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-dev@hadoop.apache.org Delivered-To: mailing list core-commits@hadoop.apache.org Received: (qmail 93453 invoked by uid 99); 17 Jul 2008 19:17:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jul 2008 12:17:45 -0700 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jul 2008 19:17:00 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 3064A23889F7; Thu, 17 Jul 2008 12:16:55 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r677692 - in /hadoop/core/branches/branch-0.18: ./ bin/ src/core/org/apache/hadoop/util/ src/docs/src/documentation/content/xdocs/ src/mapred/org/apache/hadoop/mapred/ src/test/org/apache/hadoop/mapred/ src/test/testshell/ Date: Thu, 17 Jul 2008 19:16:54 -0000 To: core-commits@hadoop.apache.org From: mahadev@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20080717191655.3064A23889F7@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: mahadev Date: Thu Jul 17 12:16:53 2008 New Revision: 677692 URL: http://svn.apache.org/viewvc?rev=677692&view=rev Log: HADOOP-3743. Fix -libjars, -files, -archives options. (Amareshwari Sriramadasu via mahadev) Added: hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/JobShell.java hadoop/core/branches/branch-0.18/src/test/org/apache/hadoop/mapred/TestJobShell.java Modified: hadoop/core/branches/branch-0.18/CHANGES.txt hadoop/core/branches/branch-0.18/bin/hadoop hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/util/GenericOptionsParser.java hadoop/core/branches/branch-0.18/src/docs/src/documentation/content/xdocs/commands_manual.xml hadoop/core/branches/branch-0.18/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/JobClient.java hadoop/core/branches/branch-0.18/src/test/testshell/ExternalMapReduce.java Modified: hadoop/core/branches/branch-0.18/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/CHANGES.txt?rev=677692&r1=677691&r2=677692&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/CHANGES.txt (original) +++ hadoop/core/branches/branch-0.18/CHANGES.txt Thu Jul 17 12:16:53 2008 @@ -762,6 +762,9 @@ HADOOP-3670. Fixes JobTracker to clear out split bytes when no longer required. (Amareshwari Sriramadasu via ddas) + HADOOP-3743. Fix -libjars, -files, -archives options to work even if + user code does not implement tools. (Amareshwari Sriramadasu via mahadev) + Release 0.17.2 - Unreleased BUG FIXES Modified: hadoop/core/branches/branch-0.18/bin/hadoop URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/bin/hadoop?rev=677692&r1=677691&r2=677692&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/bin/hadoop (original) +++ hadoop/core/branches/branch-0.18/bin/hadoop Thu Jul 17 12:16:53 2008 @@ -235,7 +235,7 @@ CLASS=org.apache.hadoop.util.VersionInfo HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS" elif [ "$COMMAND" = "jar" ] ; then - CLASS=org.apache.hadoop.util.RunJar + CLASS=org.apache.hadoop.mapred.JobShell elif [ "$COMMAND" = "distcp" ] ; then CLASS=org.apache.hadoop.tools.DistCp CLASSPATH=${CLASSPATH}:${TOOL_PATH} Modified: hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/util/GenericOptionsParser.java URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/util/GenericOptionsParser.java?rev=677692&r1=677691&r2=677692&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/util/GenericOptionsParser.java (original) +++ hadoop/core/branches/branch-0.18/src/core/org/apache/hadoop/util/GenericOptionsParser.java Thu Jul 17 12:16:53 2008 @@ -247,6 +247,7 @@ conf.set(property[i], property[i+1]); } } + conf.setBoolean("mapred.used.genericoptionsparser", true); } /** Modified: hadoop/core/branches/branch-0.18/src/docs/src/documentation/content/xdocs/commands_manual.xml URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/src/docs/src/documentation/content/xdocs/commands_manual.xml?rev=677692&r1=677691&r2=677692&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/src/docs/src/documentation/content/xdocs/commands_manual.xml (original) +++ hadoop/core/branches/branch-0.18/src/docs/src/documentation/content/xdocs/commands_manual.xml Thu Jul 17 12:16:53 2008 @@ -57,7 +57,11 @@

Following are supported by dfsadmin, fs, fsck and - job. + job. + Applications should implement + Tool to support + + GenericOptions.

Modified: hadoop/core/branches/branch-0.18/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml?rev=677692&r1=677691&r2=677692&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml (original) +++ hadoop/core/branches/branch-0.18/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml Thu Jul 17 12:16:53 2008 @@ -602,6 +602,22 @@ Hello 2
World 2

+ +

Applications can specify a comma separated list of paths which + would be present in the current working directory of the task + using the option -files. The -libjars + option allows applications to add jars to the classpaths of the maps + and reduces. The -archives allows them to pass archives + as arguments that are unzipped/unjarred and a link with name of the + jar/zip are created in the current working directory of tasks. More + details about the command line options are available at + Commands manual

+ +

Running wordcount example with + -libjars and -files:
+ hadoop jar hadoop-examples.jar wordcount -files cachefile.txt + -libjars mylib.jar input output +

Modified: hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/JobClient.java URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/JobClient.java?rev=677692&r1=677691&r2=677692&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/JobClient.java (original) +++ hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/JobClient.java Thu Jul 17 12:16:53 2008 @@ -153,6 +153,7 @@ private static final Log LOG = LogFactory.getLog("org.apache.hadoop.mapred.JobClient"); public static enum TaskStatusFilter { NONE, KILLED, FAILED, SUCCEEDED, ALL } private TaskStatusFilter taskOutputFilter = TaskStatusFilter.FAILED; + private static Configuration commandLineConfig; static long MAX_JOBPROFILE_AGE = 1000 * 2; /** @@ -359,8 +360,26 @@ setConf(conf); init(conf); } + + /** + * set the command line config in the jobclient. these are + * parameters paassed from the command line and stored in + * conf + * @param conf the configuration object to set. + */ + static synchronized void setCommandLineConfig(Configuration conf) { + commandLineConfig = conf; + } /** + * return the command line configuration + */ + public static synchronized Configuration getCommandLineConfig() { + return commandLineConfig; + } + + + /** * Connect to the default {@link JobTracker}. * @param conf the job configuration. * @throws IOException @@ -484,14 +503,51 @@ */ private void configureCommandLineOptions(JobConf job, Path submitJobDir, Path submitJarFile) throws IOException { + + final String warning = "Use genericOptions for the option "; + + if (!(job.getBoolean("mapred.used.genericoptionsparser", false))) { + LOG.warn("Use GenericOptionsParser for parsing the arguments. " + + "Applications should implement Tool for the same."); + } + // get all the command line arguments into the // jobconf passed in by the user conf + Configuration commandConf = JobClient.getCommandLineConfig(); String files = null; String libjars = null; String archives = null; + files = job.get("tmpfiles"); + if (files == null) { + if (commandConf != null) { + files = commandConf.get("tmpfiles"); + if (files != null) { + LOG.warn(warning + "-files"); + } + } + } + libjars = job.get("tmpjars"); + if (libjars == null) { + if (commandConf != null) { + libjars = commandConf.get("tmpjars"); + if (libjars != null) { + LOG.warn(warning + "-libjars"); + } + } + } + archives = job.get("tmparchives"); + if (archives == null) { + if (commandConf != null) { + archives = commandConf.get("tmparchives"); + if (archives != null) { + LOG.warn(warning + "-archives"); + } + } + } + /* * set this user's id in job configuration, so later job files can be * accessed using this user's id Added: hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/JobShell.java URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/JobShell.java?rev=677692&view=auto ============================================================================== --- hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/JobShell.java (added) +++ hadoop/core/branches/branch-0.18/src/mapred/org/apache/hadoop/mapred/JobShell.java Thu Jul 17 12:16:53 2008 @@ -0,0 +1,70 @@ +/* Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.mapred; + +import java.io.IOException; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.conf.Configured; +import org.apache.hadoop.util.RunJar; +import org.apache.hadoop.util.StringUtils; +import org.apache.hadoop.util.Tool; +import org.apache.hadoop.util.ToolRunner; + +/** Provide command line parsing for JobSubmission + * job submission looks like + * hadoop jar -libjars -archives + * -files inputjar args + */ +public class JobShell extends Configured implements Tool { + public JobShell() {this(null);}; + + public JobShell(Configuration conf) { + super(conf); + } + + protected void init() throws IOException { + getConf().setQuietMode(false); + } + + /** + * run method from Tool + */ + public int run(String argv[]) throws Exception { + int exitCode = -1; + Configuration conf = getConf(); + try{ + JobClient.setCommandLineConfig(conf); + try { + RunJar.main(argv); + exitCode = 0; + } catch(Throwable th) { + System.err.println(StringUtils.stringifyException(th)); + } + } catch(RuntimeException re) { + exitCode = -1; + System.err.println(re.getLocalizedMessage()); + } + return exitCode; + } + + public static void main(String[] argv) throws Exception { + JobShell jshell = new JobShell(); + ToolRunner.run(jshell, argv); + } +} Added: hadoop/core/branches/branch-0.18/src/test/org/apache/hadoop/mapred/TestJobShell.java URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/src/test/org/apache/hadoop/mapred/TestJobShell.java?rev=677692&view=auto ============================================================================== --- hadoop/core/branches/branch-0.18/src/test/org/apache/hadoop/mapred/TestJobShell.java (added) +++ hadoop/core/branches/branch-0.18/src/test/org/apache/hadoop/mapred/TestJobShell.java Thu Jul 17 12:16:53 2008 @@ -0,0 +1,79 @@ +/* Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.mapred; + +import java.io.File; +import java.io.FileOutputStream; + +import junit.framework.TestCase; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.*; +import org.apache.hadoop.util.ToolRunner; +import org.apache.hadoop.hdfs.MiniDFSCluster; + +public class TestJobShell extends TestCase { + // Input output paths for this.. + // these are all dummy and does not test + // much in map reduce except for the command line + // params + static final Path input = new Path("/test/input/"); + static final Path output = new Path("/test/output"); + File buildDir = new File(System.getProperty("test.build.data", "/tmp")); + public void testJobShell() throws Exception { + MiniDFSCluster dfs = null; + MiniMRCluster mr = null; + FileSystem fs = null; + Path testFile = new Path(input, "testfile"); + try { + Configuration conf = new Configuration(); + //start the mini mr and dfs cluster. + dfs = new MiniDFSCluster(conf, 2 , true, null); + fs = dfs.getFileSystem(); + FSDataOutputStream stream = fs.create(testFile); + stream.write("teststring".getBytes()); + stream.close(); + mr = new MiniMRCluster(2, fs.getUri().toString(), 1); + JobConf jconf = mr.createJobConf(); + JobShell jshell = new JobShell(); + File thisbuildDir = new File(buildDir, "jobCommand"); + assertTrue("create build dir", thisbuildDir.mkdirs()); + File f = new File(thisbuildDir, "files_tmp"); + FileOutputStream fstream = new FileOutputStream(f); + fstream.write("somestrings".getBytes()); + fstream.close(); + String[] args = new String[8]; + args[0] = "-files"; + args[1] = f.toString(); + args[2] = "-libjars"; + /// the testjob.jar as a temporary jar file + // rather than creating its own + args[3] = "build/test/testjar/testjob.jar"; + args[4] = "build/test/testshell/testshell.jar"; + args[5] = "testshell.ExternalMapReduce"; + args[6] = input.toString(); + args[7] = output.toString(); + int ret = ToolRunner.run(jconf, jshell, args); + assertTrue("not failed ", ret != -1); + f.delete(); + thisbuildDir.delete(); + } finally { + if (dfs != null) {dfs.shutdown();}; + if (mr != null) {mr.shutdown();}; + } + } +} Modified: hadoop/core/branches/branch-0.18/src/test/testshell/ExternalMapReduce.java URL: http://svn.apache.org/viewvc/hadoop/core/branches/branch-0.18/src/test/testshell/ExternalMapReduce.java?rev=677692&r1=677691&r2=677692&view=diff ============================================================================== --- hadoop/core/branches/branch-0.18/src/test/testshell/ExternalMapReduce.java (original) +++ hadoop/core/branches/branch-0.18/src/test/testshell/ExternalMapReduce.java Thu Jul 17 12:16:53 2008 @@ -115,8 +115,13 @@ } public static void main(String[] args) throws Exception { - int res = ToolRunner.run(new Configuration(), - new ExternalMapReduce(), args); - System.exit(res); + Configuration commandConf = JobClient.getCommandLineConfig(); + if (commandConf != null) { + ToolRunner.run(new Configuration(commandConf), + new ExternalMapReduce(), args); + } else { + ToolRunner.run(new Configuration(), + new ExternalMapReduce(), args); + } } }
GENERIC_OPTION Description