hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Maron (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
Date Mon, 28 Mar 2016 13:09:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214170#comment-15214170

Jonathan Maron commented on YARN-4757:

Let me respond with a deployment option that would address the concern, though I doubt we
can try address your security concerns given the inherent nature of DNS:

Given the scenario you describe, I believe a more viable/secure approach would be to:

1)  Deploy a load balancer fronting the web services
2)  Web services would come up as YARN components, assigned ports that are available on the
given hosts (not necessarily 9999) and advertising their address leveraging the registry and
DNS service
3)  It would be up to the load balancer to discover the available hosts (lookup name, test
connection etc), based on configuration (e.g. the names of components to lookup etc)

The load balancer is on a trusted host and is configured "outside" the YARN cluster, so the
opportunity for a "spoofing" server is mitigated.  For a given deployment, the location (SRV
record) of the load balancer could be statically defined in a DNS zone file that is configured
as the initial configuration file for the DNS service.  This scheme would require some configuration
to come up with appropriate TTL values etc. 

Interestingly enough, I've had some discussion in the past with some of the Knox developers
about its potential role in the management of connections to dynamically deployed cluster

> [Umbrella] Simplified discovery of services via DNS mechanisms
> --------------------------------------------------------------
>                 Key: YARN-4757
>                 URL: https://issues.apache.org/jira/browse/YARN-4757
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Jonathan Maron
>         Attachments: YARN-4757- Simplified discovery of services via DNS mechanisms.pdf
> [See overview doc at YARN-4692, copying the sub-section ( to track all related
> In addition to completing the present story of service­-registry (YARN-913), we also
need to simplify the access to the registry entries. The existing read mechanisms of the YARN
Service Registry are currently limited to a registry specific (java) API and a REST interface.
In practice, this makes it very difficult for wiring up existing clients and services. For
e.g, dynamic configuration of dependent end­points of a service is not easy to implement
using the present registry­-read mechanisms, *without* code-changes to existing services.
> A good solution to this is to expose the registry information through a more generic
and widely used discovery mechanism: DNS. Service Discovery via DNS uses the well-­known
DNS interfaces to browse the network for services. YARN-913 in fact talked about such a DNS
based mechanism but left it as a future task. (Task) Having the registry information exposed
via DNS simplifies the life of services.

This message was sent by Atlassian JIRA

View raw message