From yarn-issues-return-144496-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Tue May 8 22:26:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9B61418063B for ; Tue, 8 May 2018 22:26:04 +0200 (CEST) Received: (qmail 15054 invoked by uid 500); 8 May 2018 20:26:03 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 15042 invoked by uid 99); 8 May 2018 20:26:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 May 2018 20:26:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 370F01A2ED1 for ; Tue, 8 May 2018 20:26:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -102.311 X-Spam-Level: X-Spam-Status: No, score=-102.311 tagged_above=-999 required=6.31 tests=[RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id f4tsl8iSzCh6 for ; Tue, 8 May 2018 20:26:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 4B44A5F5FA for ; Tue, 8 May 2018 20:26:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 6B48BE12B6 for ; Tue, 8 May 2018 20:26:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 149F121299 for ; Tue, 8 May 2018 20:26:00 +0000 (UTC) Date: Tue, 8 May 2018 20:26:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-8256) Pluggable provider for node membership management MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-8256?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16467= 929#comment-16467929 ]=20 ASF GitHub Bot commented on YARN-8256: -------------------------------------- GitHub user functicons opened a pull request: https://github.com/apache/hadoop/pull/384 YARN-8256. Support pluggable node membership manager in YARN. [YARN-8256](https://issues.apache.org/jira/browse/YARN-8256): Support p= luggable node membership manager in YARN. You can merge this pull request into a Git repository by running: $ git pull https://github.com/functicons/hadoop trunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/384.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #384 =20 ---- commit 57a92fc8712d4fb4c6b88dc390baa2c5b30471ca Author: Dagang Wei Date: 2018-05-07T20:41:59Z Support YARN pluggable node membership. ---- > Pluggable provider for node membership management > ------------------------------------------------- > > Key: YARN-8256 > URL: https://issues.apache.org/jira/browse/YARN-8256 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager > Affects Versions: 2.8.3, 3.0.2 > Reporter: Dagang Wei > Priority: Major > Original Estimate: 24h > Remaining Estimate: 24h > > h1. Background > HDFS-7541 introduced a pluggable provider framework for node membership m= anagement, which gives HDFS the flexibility to have different ways to manag= e node membership for different needs. > [org.apache.hadoop.hdfs.server.blockmanagement.HostConfigManager|https://= github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/mai= n/java/org/apache/hadoop/hdfs/server/blockmanagement/HostConfigManager.java= ] is the class which provides the abstraction. Currently, there are 2 imple= mentations in the HDFS codebase: > 1) [org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager|https:/= /github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/ma= in/java/org/apache/hadoop/hdfs/server/blockmanagement/HostFileManager.java]= which uses 2 config files which are defined by the properties dfs.hosts an= d dfs.hosts.exclude. > 2) [org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager= |https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdf= s/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/CombinedHostF= ileManager.java] which uses a single JSON file defined by the property dfs.= hosts. > dfs.namenode.hosts.provider.classname is the property determining which i= mplementation is used > h1. Problem > YARN should be consistent with HDFS in terms of pluggable provider for no= de membership management. The absence of it makes YARN impossible to have o= ther config sources, e.g., ZooKeeper, database, other config file formats, = etc. > h1. Proposed solution > [org.apache.hadoop.yarn.server.resourcemanager.NodesListManager|https://g= ithub.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-y= arn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hado= op/yarn/server/resourcemanager/NodesListManager.java]=C2=A0is the class=C2= =A0for managing YARN node membership today. It uses [HostsFileReader|https:= //github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/s= rc/main/java/org/apache/hadoop/util/HostsFileReader.java]=C2=A0to read conf= ig files specified by the property yarn.resourcemanager.nodes.include-path = for nodes to include and yarn.resourcemanager.nodes.nodes.exclude-path for = nodes to exclude. > The proposed solution is to > 1) introduce a new interface {color:#008000}HostsConfigManager{color} whi= ch provides the abstraction for node membership management. Update {color:#= 008000}NodeListManager{color} to depend on {color:#008000}HostsConfigManage= r{color} instead of {color:#008000}HostsFileReader{color}. Then create a wr= apper class for {color:#008000}HostsFileReader{color} which implements the = interface. > 2) introduce a new config property {color:#008000}yarn.resourcemanager.ho= sts-config.manager.class{color} for specifying the implementation class. Se= t the default value to the wrapper class of {color:#008000}HostsFileReader{= color} for backward compatibility between new code and old config. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: yarn-issues-help@hadoop.apache.org