hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4948) Support node labels store in zookeeper
Date Fri, 02 Sep 2016 00:28:22 GMT

    [ https://issues.apache.org/jira/browse/YARN-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15457059#comment-15457059

Wangda Tan commented on YARN-4948:

Sorry for the late response, I just discussed with [~subru], and talked to [~naganarasimha_gr@apache.org]
few months before.

Thanks for taking up this JIRA, [~wjlei], I would suggest:
- If you want it works with WASB file system (or other file system doesn't support append),
you can try to use NonAppendableFSNodeLabelStore.  
- If you don't want YARN fails because of HDFS failures, you can setup yarn.node-labels.fs-store.retry-policy-spec

- Reasonable number of nodes and reasonable frequency of node label updating is already supported
by existing stores. There will be a lot of issue if #nodemanager goes beyond 20000+, for example,
scheduler itself becomes a bottleneck.

I would say adding the new ZK store impl will add extra overhead to code maintenance. And
since pluggable node label store is supported, if you really want to run a ZK store, you can
configure it in yarn-site.xml. So I suggest to keep this open until this becomes a common
requirement for node label.

> Support node labels store in zookeeper
> --------------------------------------
>                 Key: YARN-4948
>                 URL: https://issues.apache.org/jira/browse/YARN-4948
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: jialei weng
>            Assignee: jialei weng
>         Attachments: YARN-4948.001.patch, YARN-4948.002.patch, YARN-4948.003.patch, YARN-4948.006.patch,
> Support node labels store in zookeeper. The main scenario for this is to give a way to
decouple yarn with HDFS. Since nodelabel is a very important data for yarn, if hdfs down,
yarn will fail to start up,too. So it is meaningful for make yarn much independence when user
serve both yarn and HDFS. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message