hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Kozlov <ale...@cloudera.com>
Subject Re: When is mapred-site.xml read?
Date Tue, 21 Jun 2011 13:37:50 GMT
Hi John, You are right: the *-site.xml files are read by daemons on
startup.  However, the job's tasks are executed in a separate JVM and some
of the parameters, like max heap from *mapred.java.child.opts*, are set
during the job execution.  In this case the parameter is coming from the
client side where the whole configuration is serialized and passed to the
slave TT nodes.  It is never read from the TT node's *-site.xml.  I don't
know  a comprehensive list of parameters that can be changed by the client
side, but most of the parameters are set during the daemon startup.

On Tue, Jun 21, 2011 at 5:42 AM, John Armstrong <john.armstrong@ccri.com>wrote:

> One of my colleagues and I have a little confusion between us as to
> exactly when mapred-site.xml is read.  The pages on hadoop.apache.orgdon't
> seem to specify it very clearly.
>
> One position is that mapred-site.xml is read by the daemon processes at
> startup, and so changing a parameter in mapred-site.xml requires bouncing
> the daemons.
>
> The other position is that an individual job's launcher reads the local
> mapred-site.xml when creating its baseline configuration, so changing a
> parameter in mapred-site.xml will take effect immediately with the next job
> launched.
>
> Thanks for any help you can give improving our understanding of how these
> configuration files work.
>

Mime
View raw message