hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Masatake Iwasaki (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3749) We should make a copy of configuration when init MiniYARNCluster with multiple RMs
Date Tue, 02 Jun 2015 02:49:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568429#comment-14568429
] 

Masatake Iwasaki commented on YARN-3749:
----------------------------------------

Thanks for working on this, [~chenchun]. I would like this fix to comes in because it seems
to affect YARN-3752 I'm looking into.

{quote}
2. Do we really need the following change at MiniYARNCluster#serviceInit

   conf.set(YarnConfiguration.RM_HA_ID, "rm0");

Because I saw initResourceManager will also configure RM_HA_ID.
{quote}

When I tried similar to the patch, I got error below because {{HAUtil#getRMHAId}} called from
{{YarnConfiguration#updateConnectAddr}} expects that there is at most 1 RM id matching to
the node.

{noformat}
  2015-06-02 10:14:23,648 INFO  [Thread-284] service.AbstractService (AbstractService.java:noteFailure(272))
- Service org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
failed in state STARTED; cause: org.apache.hadoop.HadoopIllegalArgumentException: The HA Configuration
has multiple addresses that match local node's address.
  org.apache.hadoop.HadoopIllegalArgumentException: The HA Configuration has multiple addresses
that match local node's address.
          at org.apache.hadoop.yarn.conf.HAUtil.getRMHAId(HAUtil.java:204)
          at org.apache.hadoop.yarn.conf.YarnConfiguration.updateConnectAddr(YarnConfiguration.java:1971)
          at org.apache.hadoop.conf.Configuration.updateConnectAddr(Configuration.java:2129)
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceStart(ResourceLocalizationService.java:357)
          at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
          at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
          at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:467)
          at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
          at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
          at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:321)
          at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
          at org.apache.hadoop.yarn.server.MiniYARNCluster$NodeManagerWrapper$1.run(MiniYARNCluster.java:562)
{noformat}

The check can be bypassed by setting dummy value to {{yarn.resourcemanager.ha.id}} in configuration
*used by NodeManager instance*. I think there should be a comment explain that it is a dummy
for unit test at least.


> We should make a copy of configuration when init MiniYARNCluster with multiple RMs
> ----------------------------------------------------------------------------------
>
>                 Key: YARN-3749
>                 URL: https://issues.apache.org/jira/browse/YARN-3749
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Chun Chen
>            Assignee: Chun Chen
>         Attachments: YARN-3749.2.patch, YARN-3749.patch
>
>
> When I was trying to write a test case for YARN-2674, I found DS client trying to connect
to both rm1 and rm2 with the same address 0.0.0.0:18032 when RM failover. But I initially
set yarn.resourcemanager.address.rm1=0.0.0.0:18032, yarn.resourcemanager.address.rm2=0.0.0.0:28032
 After digging, I found it is in ClientRMService where the value of yarn.resourcemanager.address.rm2
changed to 0.0.0.0:18032. See the following code in ClientRMService:
> {code}
> clientBindAddress = conf.updateConnectAddr(YarnConfiguration.RM_BIND_HOST,
>                                                YarnConfiguration.RM_ADDRESS,
>                                                YarnConfiguration.DEFAULT_RM_ADDRESS,
>                                                server.getListenerAddress());
> {code}
> Since we use the same instance of configuration in rm1 and rm2 and init both RM before
we start both RM, we will change yarn.resourcemanager.ha.id to rm2 during init of rm2 and
yarn.resourcemanager.ha.id will become rm2 during starting of rm1.
> So I think it is safe to make a copy of configuration when init both of the rm.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message