hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Maron (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4757) [Umbrella] Simplified discovery of services via DNS mechanisms
Date Mon, 06 Jun 2016 12:49:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316435#comment-15316435

Jonathan Maron commented on YARN-4757:

[~jianhe] - Thanks for the review!  Answers:

Here, it’s using the ‘description’ filed for constructing the DNS name. is this expected
? seems not mentioned in the doc

The description field is currently the service record field that Slider (the current primary
user of the ZK registry) is leveraging to relate the component name (role).  I assume the
reason is because the ZK path to the node relates the user, application, and component name.
 [~steve_l] - is that correct?  Any objection to an explicit "name" attribute in the service

we can close the CuratorService#treeCache when stop the service?

Probably makes sense.

only readLock is being used. wondering whether these locks are needed.

This is probably due to some refactorings - it was leveraged in previous iterations of the
code.  I think I will re-introduce it.  I see nothing in the dnsjava code to indicate that
Zone implementations are thread safe.  Given the dynamic nature of record registration and
deletion in the yarn use case I think it would be base to synchronize access to the zone object
to ensure deterministic results.

question about the dnsEnabled config, if the dnsEnabled is false, what else does the RegistryDNSServer
do ? Asking this because I'm wondering whether this config is actually needed.

I guess this depends on what our expectations are regarding the use of DNS - is it expected
to be available as a default service.  If the answer is no, we could manage the inclusion
of the service at this level or perhaps one level (have the RM not even add the service based
on flag value?)

RecordCreatorFactory: The RecordCreatroFactory#getRecordCreator is instantiating the creator
instance every time this method gets called. May be singleton pattern could be useful to avoid
creating a new instance every time.

Certainly a possibility.  The thinking here was to create simple, lightweight, stateless objects
that could be used with little regard to multi-threading concerns etc.  However, if some profiling
indicates an issue a singleton approach may be preferable.

DNSManagementOperations class is not used anywhere , can be removed?

Yes - probably a leftover from a previous code iteration.

a few unused methods in RegistryDNS, e.g. addDSRecord, signZones. is this intended ?

For the time being - yes.  DS records appear to play a role in some DNS negative response
processing.  Though we have made strides in better support for negative responses (NXT records),
it was still somewhat unclear whether we ultimately would need to enhance support with full
DS record capabilities.  So I have left these methods in place until such time that I could
make a better determination.

what does the RegistryDNS#primaryDNS do ?

The idea here was to support a use case in which the YARN DNS server was designated as a resolver
for a suite of hosts in the cluster.  In those instances, queries that it itself could not
resolve would have to be forwarded to a "primary" DNS server for resolution.  I now think
this probably less likely, so we could certainly look at removing that feature.

> [Umbrella] Simplified discovery of services via DNS mechanisms
> --------------------------------------------------------------
>                 Key: YARN-4757
>                 URL: https://issues.apache.org/jira/browse/YARN-4757
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Jonathan Maron
>         Attachments: 0001-YARN-4757-Initial-code-submission-for-DNS-Service.patch, YARN-4757-
Simplified discovery of services via DNS mechanisms.pdf, YARN-4757-YARN-4757.001.patch, YARN-4757-YARN-4757.002.patch,
> [See overview doc at YARN-4692, copying the sub-section ( to track all related
> In addition to completing the present story of service­-registry (YARN-913), we also
need to simplify the access to the registry entries. The existing read mechanisms of the YARN
Service Registry are currently limited to a registry specific (java) API and a REST interface.
In practice, this makes it very difficult for wiring up existing clients and services. For
e.g, dynamic configuration of dependent end­points of a service is not easy to implement
using the present registry­-read mechanisms, *without* code-changes to existing services.
> A good solution to this is to expose the registry information through a more generic
and widely used discovery mechanism: DNS. Service Discovery via DNS uses the well-­known
DNS interfaces to browse the network for services. YARN-913 in fact talked about such a DNS
based mechanism but left it as a future task. (Task) Having the registry information exposed
via DNS simplifies the life of services.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message