hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saptarshi Guha" <saptarshi.g...@gmail.com>
Subject Re: Problem loading hadoop-site.xml - dumping parameters
Date Mon, 05 Jan 2009 20:50:42 GMT
For some strange reason, neither hadoop-default.xml nor
hadoop-site.xml is loading. Both files are in hadoop-0.19.0/conf/
folder.
HADOOP_CONF_DIR points to thi
In some java code, (not a job), i did
	Configuration cf = new Configuration();
	String[] xu= new
String[]{"io.file.buffer.size","io.sort.mb","io.sort.factor","mapred.tasktracker.map.tasks.maximum","hadoop.logfile.count","hadoop.logfile.size","hadoop.tmp.dir","dfs.replication","mapred.map.tasks","mapred.job.tracker","fs.default.name"};
	for(String x : xu){
	    LOG.info(x+": "+cf.get(x));
	}

All the values returned were default values(as noted in
hadoop-default.xml). Changing the value in hadoop-default.xml were not
reflected here. Grepping the source reveal  the defaults are
hardcoded. So it seems neither is hadoop-default.xml not
hadooop-site.xml loading.
Also, Configuration.java mentionds that hadoop-site.xml is depecated
(thought still loaded)

Any suggestions?
Regards
Saptarshi


On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha <saptarshi.guha@gmail.com> wrote:
> Hello,
> I have set my HADOOP_CONF_DIR to the conf folder and still not
> loading. I have to manually set the options when I create my conf.
> Have you resolved this?
>
> Regards
> Saptarshi
>
> On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss <g00dn3ss@gmail.com> wrote:
>> Hey all,
>>
>> I have a similar issue.  I am specifically having problems with the config
>> option "mapred.child.java.opts."  I set it to -Xmx1024m and it uses -Xmx200m
>> regardless.  I am running Hadoop 0.18.2 and I'm pretty sure this option was
>> working in the previous versions of Hadoop I was using.
>>
>> I am not explicitly setting HADOOP_CONF_DIR.  My site config is in
>> ${HADOOP_HOME}/conf.  Just to test things further, I wrote a small map task
>> to print out the ENV values and it has the correct value for HADOOP_HOME,
>> HADOOP_LOG_DIR, HADOOP_OPTS, etc...  I also printed out the key/values in
>> the JobConf passed to the mapper and it has my specified values for
>> fs.default.name and mapred.job.tracker.  Other settings like dfs.name.dir,
>> dfs.data.dir, and mapred.child.java.opts do not have my values.
>>
>> Any suggestion where to look at next?
>>
>> Thanks!
>>
>>
>>
>> On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
>> amarsri@yahoo-inc.com> wrote:
>>
>>> Saptarshi Guha wrote:
>>>
>>>> Hello,
>>>> I had previously emailed regarding heap size issue and have discovered
>>>> that the hadoop-site.xml is not loading completely, i.e
>>>>  Configuration defaults = new Configuration();
>>>>        JobConf jobConf = new JobConf(defaults, XYZ.class);
>>>>        System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
>>>>        System.out.println("2:"+jobConf.get("mapred.map.tasks"));
>>>>        System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
>>>>
>>>>  System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
>>>>
>>>> returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
>>>> hadoop-site.xml are very different.
>>>>
>>>> Is there a way for hadoop to dump the parameters read in from
>>>> hadoop-site.xml and hadoop-default.xml?
>>>>
>>>>
>>>>
>>> Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
>>> http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
>>>
>>> -Amareshwari
>>>
>>
>
>
>
> --
> Saptarshi Guha - saptarshi.guha@gmail.com
>



-- 
Saptarshi Guha - saptarshi.guha@gmail.com

Mime
View raw message