hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From s..@apache.org
Subject [33/50] [abbrv] hadoop git commit: YARN-7598. Document how to use classpath isolation for aux-services in YARN. Contributed by Xuan Gong.
Date Wed, 02 May 2018 19:22:08 GMT
YARN-7598. Document how to use classpath isolation for aux-services in YARN. Contributed by
Xuan Gong.

(cherry picked from commit 56788d759f47b4b158617701f543a9dcb4df69cd)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/af70c69f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/af70c69f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/af70c69f

Branch: refs/heads/YARN-8200
Commit: af70c69fb2b58f9ec25302d218abf372adf72bfc
Parents: cc5416d
Author: Junping Du <junping_du@apache.org>
Authored: Tue Apr 24 18:29:14 2018 +0800
Committer: Junping Du <junping_du@apache.org>
Committed: Tue Apr 24 18:31:29 2018 +0800

----------------------------------------------------------------------
 .../src/site/markdown/NodeManager.md            | 49 +++++++++++++++++++-
 1 file changed, 48 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/af70c69f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManager.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManager.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManager.md
index 3261cd7..12201b9 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManager.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManager.md
@@ -87,4 +87,51 @@ Step 4.  Auxiliary services.
 
   * NodeManagers in a YARN cluster can be configured to run auxiliary services. For a completely
functional NM restart, YARN relies on any auxiliary service configured to also support recovery.
This usually includes (1) avoiding usage of ephemeral ports so that previously running clients
(in this case, usually containers) are not disrupted after restart and (2) having the auxiliary
service itself support recoverability by reloading any previous state when NodeManager restarts
and reinitializes the auxiliary service.
 
-  * A simple example for the above is the auxiliary service 'ShuffleHandler' for MapReduce
(MR). ShuffleHandler respects the above two requirements already, so users/admins don't have
do anything for it to support NM restart: (1) The configuration property **mapreduce.shuffle.port**
controls which port the ShuffleHandler on a NodeManager host binds to, and it defaults to
a non-ephemeral port. (2) The ShuffleHandler service also already supports recovery of previous
state after NM restarts.
+  * A simple example for the above is the auxiliary service 'ShuffleHandler' for MapReduce
(MR). ShuffleHandler respects the above two requirements already, so users/admins don't have
to do anything for it to support NM restart: (1) The configuration property **mapreduce.shuffle.port**
controls which port the ShuffleHandler on a NodeManager host binds to, and it defaults to
a non-ephemeral port. (2) The ShuffleHandler service also already supports recovery of previous
state after NM restarts.
+
+
+Auxiliary Service Classpath Isolation
+-------------------------------------
+
+### Introduction
+To launch auxiliary services on a NodeManager, users have to add their jar to NodeManager's
classpath directly, thus put them on the system classloader. But if multiple versions of the
plugin are present on the classpath, there is no control over which version actually gets
loaded. Or if there are any conflicts between the dependencies introduced by the auxiliary
services and the NodeManager itself, they can break the NodeManager, the auxiliary services,
or both. To solve this issue, we can instantiate auxiliary services using a classloader that
is different from the system classloader.
+
+### Configuration
+This section describes the configuration variables for aux-service classpath isolation.
+
+The following settings need to be set in *yarn-site.xml*.
+
+|Configuration Name | Description |
+|:---- |:---- |
+| `yarn.nodemanager.aux-services.%s.classpath` | Provide local directory which includes the
related jar file as well as all the dependencies’ jar file. We could specify the single
jar file or use ${local_dir_to_jar}/* to load all jars under the dep directory. |
+| `yarn.nodemanager.aux-services.%s.remote-classpath` | Provide remote absolute or relative
path to jar file(We also support zip, tar.gz, tgz, tar and gz files as well). For the same
aux-service class, we can only specify one of the configurations: yarn.nodemanager.aux-services.%s.classpath
or yarn.nodemanager.aux-services.%s.remote-classpath. The YarnRuntimeException will be thrown.
Please also make sure that the owner of the jar file must be the same as the NodeManager user
and the permbits should satisfy (permbits & 0022)==0 (such as 600, it's not writable by
group or other).|
+| `yarn.nodemanager.aux-services.%s.system-classes` | Normally, we do not need to set this
configuration. The class would be loaded from customized classpath if it does not belongs
to system-classes. For example, by default, the package org.apache.hadoop is in the system-classes,
if your class CustomAuxService is in the package org.apache.hadoop, it would not be loaded
from customized classpath. To solve this, either we could change the package for CustomAuxService,
or configure our own system-classes which exclude org.apache.hadoop. |
+
+### Configuration Examples
+
+	<property>
+		<name>yarn.nodemanager.aux-services</name>
+		<value>mapreduce_shuffle,CustomAuxService</value>
+	</property>
+
+	<property>
+		<name>yarn.nodemanager.aux-services.CustomAuxService.classpath</name>
+		<value>${local_dir_to_jar}/CustomAuxService.jar</value>
+	</property>
+
+    <!--
+	<property>
+		<name>yarn.nodemanager.aux-services.CustomAuxService.remote-classpath</name>
+		<value>${remote-dir_to_jar}/CustomAuxService.jar</value>
+	</property>
+    -->
+
+	<property>
+		<name>yarn.nodemanager.aux-services.CustomAuxService.class</name>
+		<value>org.aux.CustomAuxService</value>
+	</property>
+
+	<property>
+		<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
+		<value>org.apache.hadoop.mapred.ShuffleHandler</value>
+	</property>
\ No newline at end of file


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


Mime
View raw message