hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
Date Thu, 24 Mar 2016 16:08:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210465#comment-15210465

Robert Joseph Evans commented on YARN-4757:


Thanks for the answers.

As for the SRV records and which IP address is returned it might be good to make it more clear
in the document what you are proposing. 

Security makes since, it is almost exactly the same as what we do for storm (I really wish
ZK had delegation tokens though).

My concern was not about do we have the ability to return multiple addresses.  My concern
was mostly about how many can we return.  Typically the address returned for google.com, etc
are actually pointing to a very large load balancer and not individual web servers.  So the
number of entries returned is on the order of the number of data centers someone has, or even
more likely it is even higher level and it is around the number of geographic regions.  At
Yahoo we run very large HBase clusters.  I'm not sure how well tools would handle getting
back 2000 IP addressed for a record.  I mostly want to understand what if any theoretical
limits there are to this technology and what if any practical limits there are.

> [Umbrella] Simplified discovery of services via DNS mechanisms
> --------------------------------------------------------------
>                 Key: YARN-4757
>                 URL: https://issues.apache.org/jira/browse/YARN-4757
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Jonathan Maron
>         Attachments: YARN-4757- Simplified discovery of services via DNS mechanisms.pdf
> [See overview doc at YARN-4692, copying the sub-section ( to track all related
> In addition to completing the present story of service­-registry (YARN-913), we also
need to simplify the access to the registry entries. The existing read mechanisms of the YARN
Service Registry are currently limited to a registry specific (java) API and a REST interface.
In practice, this makes it very difficult for wiring up existing clients and services. For
e.g, dynamic configuration of dependent end­points of a service is not easy to implement
using the present registry­-read mechanisms, *without* code-changes to existing services.
> A good solution to this is to expose the registry information through a more generic
and widely used discovery mechanism: DNS. Service Discovery via DNS uses the well-­known
DNS interfaces to browse the network for services. YARN-913 in fact talked about such a DNS
based mechanism but left it as a future task. (Task) Having the registry information exposed
via DNS simplifies the life of services.

This message was sent by Atlassian JIRA

View raw message