hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4948) Support node labels store in zookeeper
Date Fri, 08 Jul 2016 13:39:11 GMT

    [ https://issues.apache.org/jira/browse/YARN-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367684#comment-15367684
] 

Naganarasimha G R commented on YARN-4948:
-----------------------------------------

Thanks for multiple patches [~wjlei],
bq. The main scenario for this is to give a way to decouple yarn with HDFS. Since nodelabel
is a very important data for yarn, if hdfs down, yarn still fail to start up. So I think this
change is meaningful for make yarn much independence when user serve both yarn and HDFS. 
Thanks for the update, Are you targeting for the use case of cloud ? In most cases HDFS and
YARN are tied up and also NodeLabelStore supports Local file system too though its not reliable.
IMO think its *not* a must have but good to have feature. So let me take a look at it.
bq. I make a way to beyond the zknode 1MB storage limitation. In UT, it can support 100000
nodemanager, recover all 100000 node nodelabel cost 4 seconds. It is really efficiency and
reliable.
I was targetting more towards Constraint labels where in approx 50 cluster constraint node
labels and decent cluster size of 5000 and each node having around 10 constraint labels. so
did some maths on it and should not impact so much.
Give me some time, Let me go over the patch and give my views/comments 

> Support node labels store in zookeeper
> --------------------------------------
>
>                 Key: YARN-4948
>                 URL: https://issues.apache.org/jira/browse/YARN-4948
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: jialei weng
>            Assignee: jialei weng
>         Attachments: YARN-4948.001.patch, YARN-4948.002.patch, YARN-4948.003.patch, YARN-4948.006.patch,
YARN-4948.007.patch
>
>
> Support node labels store in zookeeper. The main scenario for this is to give a way to
decouple yarn with HDFS. Since nodelabel is a very important data for yarn, if hdfs down,
yarn will fail to start up,too. So it is meaningful for make yarn much independence when user
serve both yarn and HDFS. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message