From common-commits-return-82318-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Fri May 4 18:38:55 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id DA494180634 for ; Fri, 4 May 2018 18:38:54 +0200 (CEST) Received: (qmail 9418 invoked by uid 500); 4 May 2018 16:38:53 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 9409 invoked by uid 99); 4 May 2018 16:38:53 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 May 2018 16:38:53 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id B9ECDF1816; Fri, 4 May 2018 16:38:53 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: eyang@apache.org To: common-commits@hadoop.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: hadoop git commit: YARN-8223. Improved yarn auxiliary service to load jar file from HDFS. Contributed by Zian Chen Date: Fri, 4 May 2018 16:38:53 +0000 (UTC) Repository: hadoop Updated Branches: refs/heads/branch-3.1 04c3f11de -> a984abc47 YARN-8223. Improved yarn auxiliary service to load jar file from HDFS. Contributed by Zian Chen (cherry picked from commit 8cdb032aff4237d8d3970057d82290e4e32c4040) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a984abc4 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a984abc4 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a984abc4 Branch: refs/heads/branch-3.1 Commit: a984abc4724fe3862e29560cfa59c5086d91c3de Parents: 04c3f11 Author: Eric Yang Authored: Fri May 4 12:36:31 2018 -0400 Committer: Eric Yang Committed: Fri May 4 12:38:28 2018 -0400 ---------------------------------------------------------------------- .../PluggableShuffleAndPluggableSort.md | 44 ++++++++++++++++++++ .../containermanager/AuxServices.java | 19 ++++++++- 2 files changed, 61 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/a984abc4/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/PluggableShuffleAndPluggableSort.md ---------------------------------------------------------------------- diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/PluggableShuffleAndPluggableSort.md b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/PluggableShuffleAndPluggableSort.md index 5ea0567..9e24103 100644 --- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/PluggableShuffleAndPluggableSort.md +++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/PluggableShuffleAndPluggableSort.md @@ -67,6 +67,50 @@ The collector class configuration may specify a comma-separated list of collecto |:---- |:---- |:---- | | `yarn.nodemanager.aux-services` | `...,mapreduce_shuffle` | The auxiliary service name | | `yarn.nodemanager.aux-services.mapreduce_shuffle.class` | `org.apache.hadoop.mapred.ShuffleHandler` | The auxiliary service class to use | +| `yarn.nodemanager.aux-services.%s.classpath` | NONE | local directory which includes the related jar file as well as all the dependencies’ jar file. We could specify the single jar file or use /dep/* to load all jars under the dep directory. | +| `yarn.nodemanager.aux-services.%s.remote-classpath` | NONE | The remote absolute or relative path to jar file | + +#### Example of loading jar file from HDFS: + +```xml + + + yarn.nodemanager.aux-services + mapreduce_shuffle,AuxServiceFromHDFS + + + + yarn.nodemanager.aux-services.AuxServiceFromHDFS.remote-classpath + /aux/test/aux-service-hdfs.jar + + + + yarn.nodemanager.aux-services.AuxServiceFromHDFS.class</name> + org.apache.auxtest.AuxServiceFromHDFS2 + + +``` + +#### Example of loading jar file from local file system: + +```xml + + + yarn.nodemanager.aux-services + mapreduce_shuffle,AuxServiceFromHDFS + + + + yarn.nodemanager.aux-services.AuxServiceFromHDFS.classpath + /aux/test/aux-service-hdfs.jar + + + + yarn.nodemanager.aux-services.AuxServiceFromHDFS.class</name> + org.apache.auxtest.AuxServiceFromHDFS2 + + +``` **IMPORTANT:** If setting an auxiliary service in addition the default `mapreduce_shuffle` service, then a new service key should be added to the http://git-wip-us.apache.org/repos/asf/hadoop/blob/a984abc4/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java index c8b7a76..3fe3cfd 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java @@ -230,15 +230,30 @@ public class AuxServices extends AbstractService } } if (reDownload) { + LocalResourceType srcType = null; + String lowerDst = StringUtils.toLowerCase(src.toString()); + if (lowerDst.endsWith(".jar")) { + srcType = LocalResourceType.FILE; + } else if (lowerDst.endsWith(".zip") || + lowerDst.endsWith(".tar.gz") || lowerDst.endsWith(".tgz") + || lowerDst.endsWith(".tar")) { + srcType = LocalResourceType.ARCHIVE; + } else { + throw new YarnRuntimeException( + "Can not unpack file from remote-file-path:" + src + + "for aux-service:" + ".\n"); + } LocalResource scRsrc = LocalResource.newInstance( URL.fromURI(src.toUri()), - LocalResourceType.ARCHIVE, LocalResourceVisibility.PRIVATE, + srcType, LocalResourceVisibility.PRIVATE, scFileStatus.getLen(), scFileStatus.getModificationTime()); FSDownload download = new FSDownload(localLFS, null, conf, downloadDest, scRsrc, null); try { Path downloaded = download.call(); - dest = new Path(downloaded + Path.SEPARATOR + "*"); + // don't need to convert downloaded path into a dir + // since its already a jar path. + dest = downloaded; } catch (Exception ex) { throw new YarnRuntimeException( "Exception happend while downloading files " --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-commits-help@hadoop.apache.org