hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From g00dn3ss <g00dn...@gmail.com>
Subject Re: Problem loading hadoop-site.xml - dumping parameters
Date Tue, 06 Jan 2009 19:37:56 GMT
OK, I figured out my problem.  As expected, this was my silly mistake.  We
have a java program that runs on a machine outside of our hadoop cluster
that references the hadoop jar files.  This machine submits jobs to our
cluster but doesn't actually have hadoop installed.  I was configuring the
mapred.child.java.opts in the hadoop-site.xml used by the cluster.  However,
the value that the tasks actually use come from the JobConf of the submitted
job.  So I am now manually setting the options I want in our external
program that submits the jobs.  It seems to be working as expected now.  I
thought my old setup was working with the older version of hadoop, but it
may be that our jobs just recently started hitting the 200m heap boundary.

Not sure if this is also your problem, Saptarshi.  But hopefully it will
lead you in the right direction.

g00dn3ss



On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha <saptarshi.guha@gmail.com>wrote:

> Hello,
> I have set my HADOOP_CONF_DIR to the conf folder and still not
> loading. I have to manually set the options when I create my conf.
> Have you resolved this?
>
> Regards
> Saptarshi
>
> On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss <g00dn3ss@gmail.com> wrote:
> > Hey all,
> >
> > I have a similar issue.  I am specifically having problems with the
> config
> > option "mapred.child.java.opts."  I set it to -Xmx1024m and it uses
> -Xmx200m
> > regardless.  I am running Hadoop 0.18.2 and I'm pretty sure this option
> was
> > working in the previous versions of Hadoop I was using.
> >
> > I am not explicitly setting HADOOP_CONF_DIR.  My site config is in
> > ${HADOOP_HOME}/conf.  Just to test things further, I wrote a small map
> task
> > to print out the ENV values and it has the correct value for HADOOP_HOME,
> > HADOOP_LOG_DIR, HADOOP_OPTS, etc...  I also printed out the key/values in
> > the JobConf passed to the mapper and it has my specified values for
> > fs.default.name and mapred.job.tracker.  Other settings like
> dfs.name.dir,
> > dfs.data.dir, and mapred.child.java.opts do not have my values.
> >
> > Any suggestion where to look at next?
> >
> > Thanks!
> >
> >
> >
> > On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
> > amarsri@yahoo-inc.com> wrote:
> >
> >> Saptarshi Guha wrote:
> >>
> >>> Hello,
> >>> I had previously emailed regarding heap size issue and have discovered
> >>> that the hadoop-site.xml is not loading completely, i.e
> >>>  Configuration defaults = new Configuration();
> >>>        JobConf jobConf = new JobConf(defaults, XYZ.class);
> >>>        System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
> >>>        System.out.println("2:"+jobConf.get("mapred.map.tasks"));
> >>>        System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
> >>>
> >>>
>  System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
> >>>
> >>> returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
> >>> hadoop-site.xml are very different.
> >>>
> >>> Is there a way for hadoop to dump the parameters read in from
> >>> hadoop-site.xml and hadoop-default.xml?
> >>>
> >>>
> >>>
> >> Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
> >>
> http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
> >>
> >> -Amareshwari
> >>
> >
>
>
>
> --
> Saptarshi Guha - saptarshi.guha@gmail.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message