hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8256) Pluggable provider for node membership management
Date Tue, 08 May 2018 20:26:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467929#comment-16467929
] 

ASF GitHub Bot commented on YARN-8256:
--------------------------------------

GitHub user functicons opened a pull request:

    https://github.com/apache/hadoop/pull/384

    YARN-8256. Support pluggable node membership manager in YARN.

    [YARN-8256](https://issues.apache.org/jira/browse/YARN-8256): Support pluggable node membership
manager in YARN.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/functicons/hadoop trunk

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/hadoop/pull/384.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #384
    
----
commit 57a92fc8712d4fb4c6b88dc390baa2c5b30471ca
Author: Dagang Wei <functicons@...>
Date:   2018-05-07T20:41:59Z

    Support YARN pluggable node membership.

----


> Pluggable provider for node membership management
> -------------------------------------------------
>
>                 Key: YARN-8256
>                 URL: https://issues.apache.org/jira/browse/YARN-8256
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: resourcemanager
>    Affects Versions: 2.8.3, 3.0.2
>            Reporter: Dagang Wei
>            Priority: Major
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> h1. Background
> HDFS-7541 introduced a pluggable provider framework for node membership management, which
gives HDFS the flexibility to have different ways to manage node membership for different
needs.
> [org.apache.hadoop.hdfs.server.blockmanagement.HostConfigManager|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostConfigManager.java]
is the class which provides the abstraction. Currently, there are 2 implementations in the
HDFS codebase:
> 1) [org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostFileManager.java]
which uses 2 config files which are defined by the properties dfs.hosts and dfs.hosts.exclude.
> 2) [org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CombinedHostFileManager.java]
which uses a single JSON file defined by the property dfs.hosts.
> dfs.namenode.hosts.provider.classname is the property determining which implementation
is used
> h1. Problem
> YARN should be consistent with HDFS in terms of pluggable provider for node membership
management. The absence of it makes YARN impossible to have other config sources, e.g., ZooKeeper,
database, other config file formats, etc.
> h1. Proposed solution
> [org.apache.hadoop.yarn.server.resourcemanager.NodesListManager|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java] is
the class for managing YARN node membership today. It uses [HostsFileReader|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java] to
read config files specified by the property yarn.resourcemanager.nodes.include-path for nodes
to include and yarn.resourcemanager.nodes.nodes.exclude-path for nodes to exclude.
> The proposed solution is to
> 1) introduce a new interface {color:#008000}HostsConfigManager{color} which provides
the abstraction for node membership management. Update {color:#008000}NodeListManager{color}
to depend on {color:#008000}HostsConfigManager{color} instead of {color:#008000}HostsFileReader{color}.
Then create a wrapper class for {color:#008000}HostsFileReader{color} which implements the
interface.
> 2) introduce a new config property {color:#008000}yarn.resourcemanager.hosts-config.manager.class{color}
for specifying the implementation class. Set the default value to the wrapper class of {color:#008000}HostsFileReader{color}
for backward compatibility between new code and old config.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message