hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saptarshi Guha" <saptarshi.g...@gmail.com>
Subject Re: Problem loading hadoop-site.xml - dumping parameters
Date Tue, 06 Jan 2009 02:12:10 GMT
Possibly. When i force load the configuration
	Configuration cf = new Configuration();
	LOG.info("Adding new resource");
       cf.addResource("hadoop-site.xml")

It doesn't load, though hadoop-site.xml is present in
hadoop-0.19.0/conf, and the hadoop-0.19 folder is replicated across
all the machines. However if load it via
	cf.addResource(new
Path("/home/godhuli/custom/hadoop-0.19.0/conf/hadoop-site.xml"));
It works! So clearly, there is some other hadoop-site.xml in the
classpath, but where? Should I add
/home/godhuli/custom/hadoop-0.19.0/conf/ to the Hadoop classpath in
hadoop-env.sh?

Thanks
Saptarshi

On Mon, Jan 5, 2009 at 5:09 PM, Jason Venner <jason@attributor.com> wrote:
> somehow you have alternate versions of the file earlier in the class path.
>
> Perhaps someone's empty copies are bundled into one of your application jar
> files.
>
> Or perhaps the configurationfiles are not distributed to the datanodes in
> the expected locations.
>
> Saptarshi Guha wrote:
>>
>> For some strange reason, neither hadoop-default.xml nor
>> hadoop-site.xml is loading. Both files are in hadoop-0.19.0/conf/
>> folder.
>> HADOOP_CONF_DIR points to thi
>> In some java code, (not a job), i did
>>        Configuration cf = new Configuration();
>>        String[] xu= new
>>
>> String[]{"io.file.buffer.size","io.sort.mb","io.sort.factor","mapred.tasktracker.map.tasks.maximum","hadoop.logfile.count","hadoop.logfile.size","hadoop.tmp.dir","dfs.replication","mapred.map.tasks","mapred.job.tracker","fs.default.name"};
>>        for(String x : xu){
>>            LOG.info(x+": "+cf.get(x));
>>        }
>>
>> All the values returned were default values(as noted in
>> hadoop-default.xml). Changing the value in hadoop-default.xml were not
>> reflected here. Grepping the source reveal  the defaults are
>> hardcoded. So it seems neither is hadoop-default.xml not
>> hadooop-site.xml loading.
>> Also, Configuration.java mentionds that hadoop-site.xml is depecated
>> (thought still loaded)
>>
>> Any suggestions?
>> Regards
>> Saptarshi
>>
>>
>> On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha <saptarshi.guha@gmail.com>
>> wrote:
>>
>>>
>>> Hello,
>>> I have set my HADOOP_CONF_DIR to the conf folder and still not
>>> loading. I have to manually set the options when I create my conf.
>>> Have you resolved this?
>>>
>>> Regards
>>> Saptarshi
>>>
>>> On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss <g00dn3ss@gmail.com> wrote:
>>>
>>>>
>>>> Hey all,
>>>>
>>>> I have a similar issue.  I am specifically having problems with the
>>>> config
>>>> option "mapred.child.java.opts."  I set it to -Xmx1024m and it uses
>>>> -Xmx200m
>>>> regardless.  I am running Hadoop 0.18.2 and I'm pretty sure this option
>>>> was
>>>> working in the previous versions of Hadoop I was using.
>>>>
>>>> I am not explicitly setting HADOOP_CONF_DIR.  My site config is in
>>>> ${HADOOP_HOME}/conf.  Just to test things further, I wrote a small map
>>>> task
>>>> to print out the ENV values and it has the correct value for
>>>> HADOOP_HOME,
>>>> HADOOP_LOG_DIR, HADOOP_OPTS, etc...  I also printed out the key/values
>>>> in
>>>> the JobConf passed to the mapper and it has my specified values for
>>>> fs.default.name and mapred.job.tracker.  Other settings like
>>>> dfs.name.dir,
>>>> dfs.data.dir, and mapred.child.java.opts do not have my values.
>>>>
>>>> Any suggestion where to look at next?
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>> On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
>>>> amarsri@yahoo-inc.com> wrote:
>>>>
>>>>
>>>>>
>>>>> Saptarshi Guha wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Hello,
>>>>>> I had previously emailed regarding heap size issue and have discovered
>>>>>> that the hadoop-site.xml is not loading completely, i.e
>>>>>>  Configuration defaults = new Configuration();
>>>>>>       JobConf jobConf = new JobConf(defaults, XYZ.class);
>>>>>>       System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
>>>>>>       System.out.println("2:"+jobConf.get("mapred.map.tasks"));
>>>>>>       System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
>>>>>>
>>>>>>
>>>>>>  System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
>>>>>>
>>>>>> returns -Xmx200m, 2,1,2 respectively, even though the numbers in
the
>>>>>> hadoop-site.xml are very different.
>>>>>>
>>>>>> Is there a way for hadoop to dump the parameters read in from
>>>>>> hadoop-site.xml and hadoop-default.xml?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR)
>>>>> directory?
>>>>>
>>>>> http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
>>>>>
>>>>> -Amareshwari
>>>>>
>>>>>
>>>
>>> --
>>> Saptarshi Guha - saptarshi.guha@gmail.com
>>>
>>>
>>
>>
>>
>>
>



-- 
Saptarshi Guha - saptarshi.guha@gmail.com

Mime
View raw message