hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-913) Add a way to register long-lived services in a YARN cluster
Date Fri, 19 Sep 2014 21:50:36 GMT

    [ https://issues.apache.org/jira/browse/YARN-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141363#comment-14141363

Allen Wittenauer commented on YARN-913:

* I have some concern around 'naked' zookeeper.* config options.  If another part of Hadoop
uses ZK, would they be expected to use the same ZK options?  e.g., what if I want a different
ZK setup for a YARN registrar vs. the NN?  

* Is there any risk about a user writing to a shared ZK with the RM? i.e., if a user kills
the ZK used for app registry through some action, what happens to the RM and other user's
bits that are running?

 * Why doesn't the hostname component allow for FQDNs? 

* Are we prepared for more backlash when another component requires working DNS? :)

* Is ZK actually the right thing to use here?
+Zookeeper has a default limit of 1MB/node. If all endpoints of a service or
+component are stored in JSON attached to that node, then there is a total limit
+of 1MB of all endpoint registration data.

If we are reducing certain fields out of fear of hitting this limit, this seems to indicate
that ZK is a bad fit and/or we are using ZK incorrectly.  It could be argued that ZK should
contain a pointer to the actual data stored on HDFS.  This allows for future growth without
having to worry about blowing ZK out.

> Add a way to register long-lived services in a YARN cluster
> -----------------------------------------------------------
>                 Key: YARN-913
>                 URL: https://issues.apache.org/jira/browse/YARN-913
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: api, resourcemanager
>    Affects Versions: 2.5.0, 2.4.1
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: 2014-09-03_Proposed_YARN_Service_Registry.pdf, 2014-09-08_YARN_Service_Registry.pdf,
RegistrationServiceDetails.txt, YARN-913-001.patch, YARN-913-002.patch, YARN-913-003.patch,
YARN-913-003.patch, YARN-913-004.patch, YARN-913-006.patch, YARN-913-007.patch, YARN-913-008.patch,
yarnregistry.pdf, yarnregistry.tla
> In a YARN cluster you can't predict where services will come up -or on what ports. The
services need to work those things out as they come up and then publish them somewhere.
> Applications need to be able to find the service instance they are to bond to -and not
any others in the cluster.
> Some kind of service registry -in the RM, in ZK, could do this. If the RM held the write
access to the ZK nodes, it would be more secure than having apps register with ZK themselves.

This message was sent by Atlassian JIRA

View raw message