hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
Date Tue, 05 Apr 2016 14:48:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15226395#comment-15226395

Allen Wittenauer commented on YARN-4757:

bq. May I ask why?

In the vast majority of cases I've seen, there's been primarily two reasons:

* performance.  A specialty, high-end DNS server (e.g., Infoblox, PowerDNS, etc) will typically
blow the doors off of home grown solutions... esp if Java GC pauses are involved. ;)  Never
mind that having forced, local resolution is a HUGE win when dealing with heavy loads or high
latency connections.  Why should I blow out my local DNS cache when I can just AXFR it to
begin with?  Latency in DNS queries matters; it's one of the reasons why servers at places
like Y! ran (and probably still do) with full grown DNS caches using BIND on the local nodes.

* limiting exposure.  It's a much simpler network and security monitoring design if the DNS
pathways are limited.  If I know that DNS server X can authoritatively answer questions for
my entire corporate network, then I know that the only DNS traffic I should be seeing are
zone transfers, especially if I also close off access to the Internet (think data center).
 If I see any extra traffic, something bad is likely going on.

bq.  If a zone key is created and properly configured, isn't the authenticity of the records

As you can see, that's mostly an irrelevant question.

You're also assuming that group A trusts group B. The folks who run the corporate DNS are
not the same people who run the data center DNS who are not the same people that run the Hadoop
systems who are not the same people who are actually using the Hadoop systems as end users.
 ANY sort of insurance policy (in this case, AFXR with bonus points for IXFR) enable a lot
more trust.

bq. Do we need to require the support of secure zone transfers?

Absolutely.  It's trivial to run out of fingers of big name companies will dismiss this totally
if they can't do resolution from their own servers.

> [Umbrella] Simplified discovery of services via DNS mechanisms
> --------------------------------------------------------------
>                 Key: YARN-4757
>                 URL: https://issues.apache.org/jira/browse/YARN-4757
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Jonathan Maron
>         Attachments: YARN-4757- Simplified discovery of services via DNS mechanisms.pdf
> [See overview doc at YARN-4692, copying the sub-section ( to track all related
> In addition to completing the present story of service­-registry (YARN-913), we also
need to simplify the access to the registry entries. The existing read mechanisms of the YARN
Service Registry are currently limited to a registry specific (java) API and a REST interface.
In practice, this makes it very difficult for wiring up existing clients and services. For
e.g, dynamic configuration of dependent end­points of a service is not easy to implement
using the present registry­-read mechanisms, *without* code-changes to existing services.
> A good solution to this is to expose the registry information through a more generic
and widely used discovery mechanism: DNS. Service Discovery via DNS uses the well-­known
DNS interfaces to browse the network for services. YARN-913 in fact talked about such a DNS
based mechanism but left it as a future task. (Task) Having the registry information exposed
via DNS simplifies the life of services.

This message was sent by Atlassian JIRA

View raw message