Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id D96B5200CDF for ; Wed, 26 Jul 2017 05:22:54 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id D6410168127; Wed, 26 Jul 2017 03:22:54 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 00C95168126 for ; Wed, 26 Jul 2017 05:22:53 +0200 (CEST) Received: (qmail 34084 invoked by uid 500); 26 Jul 2017 03:22:52 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 34075 invoked by uid 99); 26 Jul 2017 03:22:52 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jul 2017 03:22:52 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id C4BECDF9AB; Wed, 26 Jul 2017 03:22:51 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: subru@apache.org To: common-commits@hadoop.apache.org Message-Id: <56d8052e939840c2af9a0ee2ab4aa506@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: hadoop git commit: YARN-6866. Minor clean-up and fixes in anticipation of YARN-2915 merge with trunk. (Botong Huang via Subru). Date: Wed, 26 Jul 2017 03:22:51 +0000 (UTC) archived-at: Wed, 26 Jul 2017 03:22:55 -0000 Repository: hadoop Updated Branches: refs/heads/YARN-2915 d4cb18005 -> 62f1ce2a3 YARN-6866. Minor clean-up and fixes in anticipation of YARN-2915 merge with trunk. (Botong Huang via Subru). Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/62f1ce2a Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/62f1ce2a Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/62f1ce2a Branch: refs/heads/YARN-2915 Commit: 62f1ce2a3d9b0f246c14fa8f0e6eb8af5a431d51 Parents: d4cb180 Author: Subru Krishnan Authored: Tue Jul 25 20:22:45 2017 -0700 Committer: Subru Krishnan Committed: Tue Jul 25 20:22:45 2017 -0700 ---------------------------------------------------------------------- hadoop-project/pom.xml | 6 +++--- .../org/apache/hadoop/yarn/conf/YarnConfiguration.java | 4 ++-- .../hadoop/yarn/conf/TestYarnConfigurationFields.java | 4 ++++ .../router/clientrm/FederationClientInterceptor.java | 13 +++---------- .../hadoop-yarn-site/src/site/markdown/Federation.md | 8 ++++---- 5 files changed, 16 insertions(+), 19 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/62f1ce2a/hadoop-project/pom.xml ---------------------------------------------------------------------- diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index 7301e90..61d7c9b 100755 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -99,9 +99,9 @@ 1.0.0-M33 1.0.0 - 3.0.3 - 2.4.11 - 6.1.0.jre7 + 3.3.1 + 2.4.12 + 6.2.1.jre7 1.8 http://git-wip-us.apache.org/repos/asf/hadoop/blob/62f1ce2a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java index 0c0dd11..d2013bf 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java @@ -2645,7 +2645,7 @@ public class YarnConfiguration extends Configuration { ROUTER_PREFIX + "clientrm."; public static final String ROUTER_CLIENTRM_ADDRESS = - ROUTER_CLIENTRM_PREFIX + ".address"; + ROUTER_CLIENTRM_PREFIX + "address"; public static final int DEFAULT_ROUTER_CLIENTRM_PORT = 8050; public static final String DEFAULT_ROUTER_CLIENTRM_ADDRESS = "0.0.0.0:" + DEFAULT_ROUTER_CLIENTRM_PORT; @@ -2663,7 +2663,7 @@ public class YarnConfiguration extends Configuration { public static final String ROUTER_RMADMIN_PREFIX = ROUTER_PREFIX + "rmadmin."; public static final String ROUTER_RMADMIN_ADDRESS = - ROUTER_RMADMIN_PREFIX + ".address"; + ROUTER_RMADMIN_PREFIX + "address"; public static final int DEFAULT_ROUTER_RMADMIN_PORT = 8052; public static final String DEFAULT_ROUTER_RMADMIN_ADDRESS = "0.0.0.0:" + DEFAULT_ROUTER_RMADMIN_PORT; http://git-wip-us.apache.org/repos/asf/hadoop/blob/62f1ce2a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java index a439cd5..a035dc7 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java @@ -77,6 +77,10 @@ public class TestYarnConfigurationFields extends TestConfigurationFieldsBase { .add(YarnConfiguration.FEDERATION_STATESTORE_HEARTBEAT_INTERVAL_SECS); configurationPropsToSkipCompare .add(YarnConfiguration.RM_EPOCH); + configurationPropsToSkipCompare + .add(YarnConfiguration.ROUTER_CLIENTRM_ADDRESS); + configurationPropsToSkipCompare + .add(YarnConfiguration.ROUTER_RMADMIN_ADDRESS); // Federation policies configs to be ignored configurationPropsToSkipCompare http://git-wip-us.apache.org/repos/asf/hadoop/blob/62f1ce2a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java index ecf53ac..7268ebd 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java @@ -19,7 +19,6 @@ package org.apache.hadoop.yarn.server.router.clientrm; import java.io.IOException; -import java.security.PrivilegedExceptionAction; import java.util.ArrayList; import java.util.List; import java.util.Map; @@ -88,10 +87,10 @@ import org.apache.hadoop.yarn.api.protocolrecords.UpdateApplicationPriorityRespo import org.apache.hadoop.yarn.api.protocolrecords.UpdateApplicationTimeoutsRequest; import org.apache.hadoop.yarn.api.protocolrecords.UpdateApplicationTimeoutsResponse; import org.apache.hadoop.yarn.api.records.ApplicationId; -import org.apache.hadoop.yarn.client.ClientRMProxy; import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.exceptions.YarnException; import org.apache.hadoop.yarn.exceptions.YarnRuntimeException; +import org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil; import org.apache.hadoop.yarn.server.federation.policies.FederationPolicyUtils; import org.apache.hadoop.yarn.server.federation.policies.RouterPolicyFacade; import org.apache.hadoop.yarn.server.federation.policies.exceptions.FederationPolicyInitializationException; @@ -175,14 +174,8 @@ public class FederationClientInterceptor ApplicationClientProtocol clientRMProxy = null; try { - clientRMProxy = - user.doAs(new PrivilegedExceptionAction() { - @Override - public ApplicationClientProtocol run() throws Exception { - return ClientRMProxy.createRMProxy(getConf(), - ApplicationClientProtocol.class); - } - }); + clientRMProxy = FederationProxyProviderUtil.createRMProxy(getConf(), + ApplicationClientProtocol.class, subClusterId, user); } catch (Exception e) { RouterServerUtil.logAndThrowException( "Unable to create the interface to reach the SubCluster " http://git-wip-us.apache.org/repos/asf/hadoop/blob/62f1ce2a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md ---------------------------------------------------------------------- diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md index 79225b4..61cb77f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md @@ -209,9 +209,9 @@ Optional: | Property | Example | Description | |:---- |:---- | |`yarn.router.hostname` | `0.0.0.0` | Router host name. -|`yarn.router.clientrm.address` | `0.0.0.0:8032` | Router client address. | +|`yarn.router.clientrm.address` | `0.0.0.0:8050` | Router client address. | |`yarn.router.webapp.address` | `0.0.0.0:80` | Webapp address at the router. | -|`yarn.router.admin.address` | `0.0.0.0:8033` | Admin address at the router. | +|`yarn.router.admin.address` | `0.0.0.0:8052` | Admin address at the router. | |`yarn.router.webapp.https.address` | `0.0.0.0:443` | Secure webapp address at the router. | |`yarn.router.submit.retry` | `3` | The number of retries in the router before we give up. | |`yarn.federation.statestore.max-connections` | `10` | This is the maximum number of parallel connections each Router makes to the state-store. | @@ -227,7 +227,7 @@ These are extra configurations that should appear in the **conf/yarn-site.xml** |:---- |:---- | | `yarn.nodemanager.amrmproxy.enabled` | `true` | Whether or not the AMRMProxy is enabled. |`yarn.nodemanager.amrmproxy.interceptor-class.pipeline` | `org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor` | A comma-separated list of interceptors to be run at the amrmproxy. For federation the last step in the pipeline should be the FederationInterceptor. -| `yarn.client.failover.proxy-provider` | `org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider` | The class used to connect to the RMs by looking up the membership information in federation state-store. This must be set if federation is enabled, even if RM HA is not enabled.| +| `yarn.client.failover-proxy-provider` | `org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider` | The class used to connect to the RMs by looking up the membership information in federation state-store. This must be set if federation is enabled, even if RM HA is not enabled.| Optional: @@ -242,7 +242,7 @@ Currently, the only supported implementation of the state-store is Microsoft SQL Running a Sample Job -------------------- -In order to submit jobs to a Federation cluster one must create a seperate set of configs for the client from which said jobs will be submitted. In these, the **conf/yarn-site.xml** should have the following additional configurations: +In order to submit jobs to a Federation cluster one must create a seperate set of configs for the client from which jobs will be submitted. In these, the **conf/yarn-site.xml** should have the following additional configurations: | Property | Example | Description | |:--- |:--- | --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-commits-help@hadoop.apache.org