hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From g00dn3ss <g00dn...@gmail.com>
Subject Re: Problem loading hadoop-site.xml - dumping parameters
Date Tue, 30 Dec 2008 22:25:18 GMT
Hey all,

I have a similar issue.  I am specifically having problems with the config
option "mapred.child.java.opts."  I set it to -Xmx1024m and it uses -Xmx200m
regardless.  I am running Hadoop 0.18.2 and I'm pretty sure this option was
working in the previous versions of Hadoop I was using.

I am not explicitly setting HADOOP_CONF_DIR.  My site config is in
${HADOOP_HOME}/conf.  Just to test things further, I wrote a small map task
to print out the ENV values and it has the correct value for HADOOP_HOME,
HADOOP_LOG_DIR, HADOOP_OPTS, etc...  I also printed out the key/values in
the JobConf passed to the mapper and it has my specified values for
fs.default.name and mapred.job.tracker.  Other settings like dfs.name.dir,
dfs.data.dir, and mapred.child.java.opts do not have my values.

Any suggestion where to look at next?

Thanks!



On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
amarsri@yahoo-inc.com> wrote:

> Saptarshi Guha wrote:
>
>> Hello,
>> I had previously emailed regarding heap size issue and have discovered
>> that the hadoop-site.xml is not loading completely, i.e
>>  Configuration defaults = new Configuration();
>>        JobConf jobConf = new JobConf(defaults, XYZ.class);
>>        System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
>>        System.out.println("2:"+jobConf.get("mapred.map.tasks"));
>>        System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
>>
>>  System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
>>
>> returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
>> hadoop-site.xml are very different.
>>
>> Is there a way for hadoop to dump the parameters read in from
>> hadoop-site.xml and hadoop-default.xml?
>>
>>
>>
> Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
> http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
>
> -Amareshwari
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message