Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 20 May 2016 16:37:13 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12946681.1457039063000.250085.1463762233459@Atlassian.JIRA>
In-Reply-To: <JIRA.12946681.1457039063000@Atlassian.JIRA>
References: <JIRA.12946681.1457039063000@Atlassian.JIRA> <JIRA.12946681.1457039063332@arcas>
Subject: [jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of
 services via DNS mechanisms
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Fri, 20 May 2016 16:37:15 -0000


    [ https://issues.apache.org/jira/browse/YARN-4757?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15293=
665#comment-15293665 ]=20

Steve Loughran commented on YARN-4757:
--------------------------------------

I am supportive of this; we always had a goal of supporting DNS (somehow); =
our naming policy was designed for that.


A troublespot is going to be that user names, even/especially kerberos name=
s, may not be valid for DNS. AD accounts often have spaces in; i18n names e=
xist, then there is punctuation. In YARN-913 I did try using punycode, but =
that only converts high-ASCII, high-unicode chars down =E2=80=94it does not=
hing for spaces in a name.

We can control the other bits, but not that.

Related to that, the parse guidelines on p8 shouldn't have a mixedcase exam=
ple "aUser", as that my be misconstrued as case being relevant.

p12; architecture good, just bear in mind that the DNS service will need to=
 handle ZK node failure/failover, so be able to switch to a new ZK node and=
 (presumably) rebuild its state from that one. Similarly, implementation wi=
ll need to handle the startup state 'zk server not yet live'.

I think you may need to add a sequence diagram for that; essentially re-enu=
m the tree with records updated to match the new state as appropriate

h3. implementation (p22).=20

* I think you may want to use one of the guava caches for doing some cachin=
g here, it could make a big diff to ZK load in some  scenarios. Example: bo=
otstrapping an app across the cluster which looks up the HBase record.

* I just had a look at how Antonio Lain did the DNS binding in Smartfrog, w=
here the Anubis HA T-Space was used as a P2P equivalent to ZK: https://sour=
ceforge.net/p/smartfrog/svn/HEAD/tree/trunk/core/components/dns/src/org/sma=
rtfrog/services/dns/ . He used dnsjava, though also added the option to act=
ually bring up BIND: https://sourceforge.net/p/smartfrog/svn/HEAD/tree/trun=
k/core/components/dns/src/org/smartfrog/services/dns/DNSBindNamedImpl.java

I'd go with dnsjava.

Note I also wrote a prototype REST server for the registry; that's hidden i=
n slider but it could be copied over to {{yarn-registry}}; it helped find s=
ome JSON/jackson marshalling problems with field naming already. The Java r=
egistry API was all built on the option of going RESTy later, and it would =
be nice for going through Knox for remote access.

Allan: consul gets a mention in the docs. Essentially, the YARN service reg=
istry works as the repo, all that's needed is to serve this up as a differe=
nt protocol through 1+ node. Because of the ZK option, code which works dir=
ectly with ZK can work with the data without worrying about DNS, DNS cachin=
g and server failures.


> [Umbrella] Simplified discovery of services via DNS mechanisms
> --------------------------------------------------------------
>
>                 Key: YARN-4757
>                 URL: https://issues.apache.org/jira/browse/YARN-4757
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Jonathan Maron
>         Attachments: 0001-YARN-4757-Initial-code-submission-for-DNS-Servi=
ce.patch, YARN-4757- Simplified discovery of services via DNS mechanisms.pd=
f
>
>
> [See overview doc at YARN-4692, copying the sub-section (3.2.10.2) to tra=
ck all related efforts.]
> In addition to completing the present story of service=C2=AD-registry (YA=
RN-913), we also need to simplify the access to the registry entries. The e=
xisting read mechanisms of the YARN Service Registry are currently limited =
to a registry specific (java) API and a REST interface. In practice, this m=
akes it very difficult for wiring up existing clients and services. For e.g=
, dynamic configuration of dependent end=C2=ADpoints of a service is not ea=
sy to implement using the present registry=C2=AD-read mechanisms, *without*=
 code-changes to existing services.
> A good solution to this is to expose the registry information through a m=
ore generic and widely used discovery mechanism: DNS. Service Discovery via=
 DNS uses the well-=C2=ADknown DNS interfaces to browse the network for ser=
vices. YARN-913 in fact talked about such a DNS based mechanism but left it=
 as a future task. (Task) Having the registry information exposed via DNS s=
implifies the life of services.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org