hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7359) Pluggable interface for cluster membership
Date Mon, 04 Jul 2011 10:51:22 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059384#comment-13059384
] 

Steve Loughran commented on HADOOP-7359:
----------------------------------------

I played for a while with a dynamic subclass of JobConf that could be changed from CM tooling
in a live system. There are a lot of places where those configs get serialized, and it's surprisingly
hard to change things on the fly: the nodes generally assume things are static for the life
of the process. That's a hard thing to deal with in the workers


LDAP/JNDI might be a better option for central sites over JDBC. This doesn't mean that I',
a fan of JNDI, only that it's easier to write back ends to than a generic JDBC driver -and
LDAP is designed with better redundancy in from the outset.


> Pluggable interface for cluster membership
> ------------------------------------------
>
>                 Key: HADOOP-7359
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7359
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Travis Crawford
>         Attachments: HADOOP-7359.diff
>
>
> Currently Hadoop uses local files to determine cluster membership. With HDFS for example,
dfs.hosts and dfs.hosts.exclude are used.
> To enable tighter integrations cluster membership should be an interface, with the current
file-based functionality provided as the default implementation. The common case would be
no functional change, however, sites could plug an alternative implementation in, such as
pulling the machine lists from a machine database.
> DETAILS:
> Two machine lists, includes and excludes, are used to define cluster membership and state.
HostsFileReader currently handles reading these lists from files, who's names are passed in
by FSNamesystem for HDFS and JobTracker for MR.
> The proposed change is adding a HostsReader interface to common, and changing HostsFileReader
to an abstract class that functions the same as today.
> Two new classes, DFSHostsFileReader and MRHostsFileReader, extend HostsFileReader and
simply pass the appropriate file names in. These new classes are needed because config key
names live outside common.
> Two new conf keys, defaulting to the file-based readers, would be added to choose a different
hosts reader: dfs.namenode.hosts.reader.class mapreduce.jobtracker.hosts.reader.class
> Comments/suggestions? I have most of this written already but would love some feedback
on the general idea before posting the diff.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message