hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From feng lu <amuseme...@gmail.com>
Subject Re: difference between mapred-site.xml and conf/masters
Date Tue, 07 Aug 2012 01:27:57 GMT
Despite its name, the conf/masters file defines on which machines Hadoop
will start secondary NameNodes in multi-node cluster. In some case, this is
just the master machine. The primary NameNode and the JobTracker will
always be the machines on which you run the bin/start-dfs.sh and
bin/start-mapred.sh scripts, respectively (the primary NameNode and the
JobTracker will be started on the same machine if you run
bin/start-all.sh). Note that you can also start an Hadoop daemon manually
on a machine via bin/hadoop-daemon.sh start [namenode | secondarynamenode |
datanode | jobtracker | tasktracker], which will not take the conf/masters
and conf/slaves files into account.

Here are more details regarding the conf/masters file, taken from the
Hadoop HDFS user guide:

The secondary NameNode merges the fsimage and the edits log files
periodically and keeps edits log size within a limit. It is usually run on
a different machine than the primary NameNode since its memory requirements
are on the same order as the primary NameNode. The secondary NameNode is
started by bin/start-dfs.sh on the nodes specified in conf/masters file.

On Tue, Aug 7, 2012 at 8:42 AM, Karthik Kambatla <kasha@cloudera.com> wrote:

> My understanding is that conf/masters is used only by the bin/ or sbin/
> scripts - like start-*.sh, while the conf property is used by the Hadoop
> daemons running on that particular node.
> Hope that helps,
> Karthik
> On Mon, Aug 6, 2012 at 5:32 PM, Momina Khan <momina.azam@gmail.com> wrote:
> > Hi,
> > I am trying to setup a hadoop cluster and I was wondering what is
> > difference in specifying the Jobtracker IP in *mapred.job.tracker* in *
> > mapred-site.xml* and noting the same IP in *conf/masters* file? Do I need
> > to do both or just one. If I need to do both, is there a difference in
> how
> > the two are used?
> >
> > thankx
> > momina
> >

Don't Grow Old, Grow Up... :-)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message