hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Venner <ja...@attributor.com>
Subject Re: Problem loading hadoop-site.xml - dumping parameters
Date Mon, 05 Jan 2009 22:09:42 GMT
somehow you have alternate versions of the file earlier in the class path.

Perhaps someone's empty copies are bundled into one of your application 
jar files.

Or perhaps the configurationfiles are not distributed to the datanodes 
in the expected locations.

Saptarshi Guha wrote:
> For some strange reason, neither hadoop-default.xml nor
> hadoop-site.xml is loading. Both files are in hadoop-0.19.0/conf/
> folder.
> HADOOP_CONF_DIR points to thi
> In some java code, (not a job), i did
> 	Configuration cf = new Configuration();
> 	String[] xu= new
> String[]{"io.file.buffer.size","io.sort.mb","io.sort.factor","mapred.tasktracker.map.tasks.maximum","hadoop.logfile.count","hadoop.logfile.size","hadoop.tmp.dir","dfs.replication","mapred.map.tasks","mapred.job.tracker","fs.default.name"};
> 	for(String x : xu){
> 	    LOG.info(x+": "+cf.get(x));
> 	}
>
> All the values returned were default values(as noted in
> hadoop-default.xml). Changing the value in hadoop-default.xml were not
> reflected here. Grepping the source reveal  the defaults are
> hardcoded. So it seems neither is hadoop-default.xml not
> hadooop-site.xml loading.
> Also, Configuration.java mentionds that hadoop-site.xml is depecated
> (thought still loaded)
>
> Any suggestions?
> Regards
> Saptarshi
>
>
> On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha <saptarshi.guha@gmail.com> wrote:
>   
>> Hello,
>> I have set my HADOOP_CONF_DIR to the conf folder and still not
>> loading. I have to manually set the options when I create my conf.
>> Have you resolved this?
>>
>> Regards
>> Saptarshi
>>
>> On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss <g00dn3ss@gmail.com> wrote:
>>     
>>> Hey all,
>>>
>>> I have a similar issue.  I am specifically having problems with the config
>>> option "mapred.child.java.opts."  I set it to -Xmx1024m and it uses -Xmx200m
>>> regardless.  I am running Hadoop 0.18.2 and I'm pretty sure this option was
>>> working in the previous versions of Hadoop I was using.
>>>
>>> I am not explicitly setting HADOOP_CONF_DIR.  My site config is in
>>> ${HADOOP_HOME}/conf.  Just to test things further, I wrote a small map task
>>> to print out the ENV values and it has the correct value for HADOOP_HOME,
>>> HADOOP_LOG_DIR, HADOOP_OPTS, etc...  I also printed out the key/values in
>>> the JobConf passed to the mapper and it has my specified values for
>>> fs.default.name and mapred.job.tracker.  Other settings like dfs.name.dir,
>>> dfs.data.dir, and mapred.child.java.opts do not have my values.
>>>
>>> Any suggestion where to look at next?
>>>
>>> Thanks!
>>>
>>>
>>>
>>> On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
>>> amarsri@yahoo-inc.com> wrote:
>>>
>>>       
>>>> Saptarshi Guha wrote:
>>>>
>>>>         
>>>>> Hello,
>>>>> I had previously emailed regarding heap size issue and have discovered
>>>>> that the hadoop-site.xml is not loading completely, i.e
>>>>>  Configuration defaults = new Configuration();
>>>>>        JobConf jobConf = new JobConf(defaults, XYZ.class);
>>>>>        System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
>>>>>        System.out.println("2:"+jobConf.get("mapred.map.tasks"));
>>>>>        System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
>>>>>
>>>>>  System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
>>>>>
>>>>> returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
>>>>> hadoop-site.xml are very different.
>>>>>
>>>>> Is there a way for hadoop to dump the parameters read in from
>>>>> hadoop-site.xml and hadoop-default.xml?
>>>>>
>>>>>
>>>>>
>>>>>           
>>>> Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
>>>> http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
>>>>
>>>> -Amareshwari
>>>>
>>>>         
>>
>> --
>> Saptarshi Guha - saptarshi.guha@gmail.com
>>
>>     
>
>
>
>   

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message