hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Íñigo Goiri (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HDFS-14118) Use DNS to resolve Namenodes and Routers
Date Wed, 30 Jan 2019 00:01:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755491#comment-16755491

Íñigo Goiri commented on HDFS-14118:

I have some confusions about the RESOLVE_SERVICE_KEY. Do you mean to put it into core-default.xml
file? Since this config is for per cluster service discovery service, if we move it to the
common, we will have to put all of the different resolvers into the core-site.xml as well.
Right, this is kind of sticky.
I guess the point is that we would have different implementations not only per namespace but
also in YARN, HDFS, Commons, etc.
In any case, we should move {{createDomainNameResolver()}} to the commons and use {{ReflectionUtils#newInstance()}}
instead of what we have there.
In that method we can pass the conf and the conf key to instantiate from.

For the MockDomainNameResolver, since what it basically does is to return some predefined
fake values, will it make more sense for individual tests to have it so that they can apply
their own values?
I'd like to make this whole resolver self contained, for that reason is better if we have
a full unit test just for it in commons.
Then others can extend it if so.

When you say `Move the setup for DNS to a function`, do you mean all of the line 252-257 to
a separate function? This doesn't bring much benefit to the code in my opinion. Can you elaborate?
I was referring to lines 137-158.
I would make this a function (e.g., {{addDNSSettings()}}) and then in {{testResolveDomainNameUsingDNS()}},
Configuration dnsConf = new Configuration(conf);

> Use DNS to resolve Namenodes and Routers
> ----------------------------------------
>                 Key: HDFS-14118
>                 URL: https://issues.apache.org/jira/browse/HDFS-14118
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Fengnan Li
>            Assignee: Fengnan Li
>            Priority: Major
>         Attachments: DNS testing log, HDFS-14118.001.patch, HDFS-14118.002.patch, HDFS-14118.003.patch,
HDFS-14118.004.patch, HDFS-14118.005.patch, HDFS-14118.006.patch, HDFS-14118.007.patch, HDFS-14118.008.patch,
> Clients will need to know about routers to talk to the HDFS cluster (obviously), and
having routers updating (adding/removing) will have to make every client change, which is
a painful process.
> DNS can be used here to resolve the single domain name clients knows to a list of routers
in the current config. However, DNS won't be able to consider only resolving to the working
router based on certain health thresholds.
> There are some ways about how this can be solved. One way is to have a separate script
to regularly check the status of the router and update the DNS records if a router fails the
health thresholds. In this way, security might be carefully considered for this way. Another
way is to have the client do the normal connecting/failover after they get the list of routers,
which requires the change of current failover proxy provider.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message