hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Maron (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
Date Wed, 23 Mar 2016 18:55:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208950#comment-15208950

Jonathan Maron commented on YARN-4757:

a)  A records are more usable from an existing client interaction perspective.  For example,
you can use a tool such as nslookup to map from a known name to its IP.  You could potentially
leverage an SRV record in that instance, but you'd have to go into the interactive mode of
nslookup, set the type, and then perform the query - a less intuitive and well known approach.

b)  It's not a matter of managing a named.conf file as much as setting up bind to support
the dynamic update protocol (YARN containers will come up and go down and those record updates
may be relatively frequent).  In addition, the stateful complaint has more to do with the
need to synch state in multiple processes rather than rely on one source of truth.  Finally,
the security needs for an internal zone server are finite enough that, if security was the
primary driver, would make the BIND selection overkill.

c) Not familiar with manta (even initial web searches didn't seem to bring anything up)? 
If there is an open source, available solution I'd be more happy to evaluate its potential

d) I'm not sure the problem is necessarily solved.  DNS is well understood, obviously.  But
the use case here - mirroring the details of an existing ZK-based registry or, more accurately,
the state of the YARN cluster - present some requirements that perhaps can be best addressed
by a tailored solution.  Given the availability of APIs such as dnsjava etc. the approach
is not necessarily daunting from a development perspective.  As such, testing can be performed
to address security and performance concerns, though I'm not naive - I understand some issues
will not manifest till actual deployment.

> [Umbrella] Simplified discovery of services via DNS mechanisms
> --------------------------------------------------------------
>                 Key: YARN-4757
>                 URL: https://issues.apache.org/jira/browse/YARN-4757
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Jonathan Maron
>         Attachments: YARN-4757- Simplified discovery of services via DNS mechanisms.pdf
> [See overview doc at YARN-4692, copying the sub-section ( to track all related
> In addition to completing the present story of service­-registry (YARN-913), we also
need to simplify the access to the registry entries. The existing read mechanisms of the YARN
Service Registry are currently limited to a registry specific (java) API and a REST interface.
In practice, this makes it very difficult for wiring up existing clients and services. For
e.g, dynamic configuration of dependent end­points of a service is not easy to implement
using the present registry­-read mechanisms, *without* code-changes to existing services.
> A good solution to this is to expose the registry information through a more generic
and widely used discovery mechanism: DNS. Service Discovery via DNS uses the well-­known
DNS interfaces to browse the network for services. YARN-913 in fact talked about such a DNS
based mechanism but left it as a future task. (Task) Having the registry information exposed
via DNS simplifies the life of services.

This message was sent by Atlassian JIRA

View raw message