hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
Date Fri, 20 May 2016 16:37:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15293665#comment-15293665

Steve Loughran commented on YARN-4757:

I am supportive of this; we always had a goal of supporting DNS (somehow); our naming policy
was designed for that.

A troublespot is going to be that user names, even/especially kerberos names, may not be valid
for DNS. AD accounts often have spaces in; i18n names exist, then there is punctuation. In
YARN-913 I did try using punycode, but that only converts high-ASCII, high-unicode chars down
—it does nothing for spaces in a name.

We can control the other bits, but not that.

Related to that, the parse guidelines on p8 shouldn't have a mixedcase example "aUser", as
that my be misconstrued as case being relevant.

p12; architecture good, just bear in mind that the DNS service will need to handle ZK node
failure/failover, so be able to switch to a new ZK node and (presumably) rebuild its state
from that one. Similarly, implementation will need to handle the startup state 'zk server
not yet live'.

I think you may need to add a sequence diagram for that; essentially re-enum the tree with
records updated to match the new state as appropriate

h3. implementation (p22). 

* I think you may want to use one of the guava caches for doing some caching here, it could
make a big diff to ZK load in some  scenarios. Example: bootstrapping an app across the cluster
which looks up the HBase record.

* I just had a look at how Antonio Lain did the DNS binding in Smartfrog, where the Anubis
HA T-Space was used as a P2P equivalent to ZK: https://sourceforge.net/p/smartfrog/svn/HEAD/tree/trunk/core/components/dns/src/org/smartfrog/services/dns/
. He used dnsjava, though also added the option to actually bring up BIND: https://sourceforge.net/p/smartfrog/svn/HEAD/tree/trunk/core/components/dns/src/org/smartfrog/services/dns/DNSBindNamedImpl.java

I'd go with dnsjava.

Note I also wrote a prototype REST server for the registry; that's hidden in slider but it
could be copied over to {{yarn-registry}}; it helped find some JSON/jackson marshalling problems
with field naming already. The Java registry API was all built on the option of going RESTy
later, and it would be nice for going through Knox for remote access.

Allan: consul gets a mention in the docs. Essentially, the YARN service registry works as
the repo, all that's needed is to serve this up as a different protocol through 1+ node. Because
of the ZK option, code which works directly with ZK can work with the data without worrying
about DNS, DNS caching and server failures.

> [Umbrella] Simplified discovery of services via DNS mechanisms
> --------------------------------------------------------------
>                 Key: YARN-4757
>                 URL: https://issues.apache.org/jira/browse/YARN-4757
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Jonathan Maron
>         Attachments: 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757-
Simplified discovery of services via DNS mechanisms.pdf
> [See overview doc at YARN-4692, copying the sub-section ( to track all related
> In addition to completing the present story of service­-registry (YARN-913), we also
need to simplify the access to the registry entries. The existing read mechanisms of the YARN
Service Registry are currently limited to a registry specific (java) API and a REST interface.
In practice, this makes it very difficult for wiring up existing clients and services. For
e.g, dynamic configuration of dependent end­points of a service is not easy to implement
using the present registry­-read mechanisms, *without* code-changes to existing services.
> A good solution to this is to expose the registry information through a more generic
and widely used discovery mechanism: DNS. Service Discovery via DNS uses the well-­known
DNS interfaces to browse the network for services. YARN-913 in fact talked about such a DNS
based mechanism but left it as a future task. (Task) Having the registry information exposed
via DNS simplifies the life of services.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message