hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Kimball <aa...@cloudera.com>
Subject Re: RE: please help in setting hadoop
Date Fri, 27 Nov 2009 18:40:09 GMT
You've set hadoop.tmp.dir to /home/hadoop/hadoop-${user.name}.

This means that on every node, you're going to need a directory named (e.g.)
/home/hadoop/hadoop-root/, since it seems as though you're running things as
root (in general, not a good policy; but ok if you're on EC2 or something
like that).

mapred.local.dir defaults to ${hadoop.tmp.dir}/mapred/local. You've
confirmed that this exists on the machine named 'master' -- what about on
slave?

Then, what are the permissions of /home/hadoop/ on the slave node? Whichever
user started the Hadoop daemons (probably either 'root' or 'hadoop') will
need the ability to mkdir /home/hadoop/hadoop-root underneath of
/home/hadoop. If that directory doesn't exist, or is chown'd to someone
else, this will probably be the result.

- Aaron


On Thu, Nov 26, 2009 at 10:22 PM, <aa225@buffalo.edu> wrote:

> Hi,
>   There should be a folder called as logs in $HADOOP_HOME. Also try going
> through
>
> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29
> .
>
>
> This is a pretty good tutorial
>
> Abhishek Agrawal
>
> SUNY- Buffalo
> (716-435-7122)
>
> On Fri 11/27/09  1:18 AM , "Krishna Kumar" krishna.kumar@nechclst.in sent:
> > I have tried, but didn't get any success. In bwt can you please tell
> exact
> > path of log file which I have to refer.
> >
> >
> > Thanks and Best Regards,
> >
> > Krishna Kumar
> >
> > Senior Storage Engineer
> >
> > Why do we have to die? If we had to die, and everything is gone after
> that,
> > then nothing else matters on this earth - everything is temporary, at
> least
> > relative to me.
> >
> >
> >
> >
> > -----Original Message-----
> >
> > From: aa225@buffalo.edu [aa225@buffa
> > lo.edu]
> > Sent: Friday, November 27, 2009 10:56 AM
> >
> > To: common-user@hadoop.apache.org
> > Subject: Re: please help in setting hadoop
> >
> >
> >
> > Hi,
> >
> > Just a thought, but you do not need to setup the temp directory in
> >
> > conf/hadoop-site.xml especially if you are running basic examples. Give
> > that a
> > shot, maybe it will work out. Otherwise see if you can find additional
> info
> > in
> > the LOGS
> >
> >
> >
> > Thank You
> >
> >
> >
> > Abhishek Agrawal
> >
> >
> >
> > SUNY- Buffalo
> >
> > (716-435-7122)
> >
> >
> >
> > On Fri 11/27/09 12:20 AM , "Krishna Kumar" kri
> > shna.kumar@nechclst.in sent:
> > > Dear All,
> >
> > > Can anybody please help me in getting out from
> > these error messages:
> > > [ hadoop]# hadoop jar
> >
> > >
> > /usr/lib/hadoop/hadoop-0.18.3-14.cloudera.CH0_3-examples.jar
> > > wordcount
> >
> > > test test-op
> >
> > >
> >
> > > 09/11/26 17:15:45 INFO mapred.FileInputFormat:
> > Total input paths to
> > > process : 4
> >
> > >
> >
> > > 09/11/26 17:15:45 INFO mapred.FileInputFormat:
> > Total input paths to
> > > process : 4
> >
> > >
> >
> > > org.apache.hadoop.ipc.RemoteException:
> > java.io.IOException: No valid
> > > local directories in property: mapred.local.dir
> >
> > >
> >
> > > at
> >
> > >
> > org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:730
> > > )
> >
> > >
> >
> > > at
> >
> > >
> > org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:222)
> > >
> >
> > > at
> >
> > >
> > org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:194)
> > >
> >
> > > at
> >
> > >
> > org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1557)
> > >
> >
> > > at
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > Method)
> >
> > >
> >
> > > at
> >
> > >
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> > > a:39)
> >
> > >
> >
> > > at
> >
> > >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> > > Impl.java:25)
> >
> > >
> >
> > > at
> > java.lang.reflect.Method.invoke(Method.java:585)
> > >
> >
> > > at
> > org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> > >
> >
> > > at
> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:890)
> > > I am running the hadoop cluster as root user on
> > two server nodes:
> > > master
> >
> > > and slave.  My hadoop-site.xml file format is as
> > follows :
> > > fs.default.name
> >
> > >
> >
> > > hdfs://master:54310
> > > dfs.permissions
> >
> > >
> >
> > > false
> >
> > > dfs.name.dir
> >
> > >
> >
> > > /home/hadoop/dfs/name
> >
> > > Further the o/p of ls command is as follows:
> >
> > >
> >
> > > [ hadoop]# ls -l /home/hadoop/hadoop-root/
> >
> > >
> >
> > > total 8
> >
> > >
> >
> > > drwxr-xr-x 4 root root 4096 Nov 26 16:48 dfs
> >
> > >
> >
> > > drwxr-xr-x 3 root root 4096 Nov 26 16:49 mapred
> >
> > >
> >
> > > [ hadoop]#
> >
> > >
> >
> > > [ hadoop]#
> >
> > >
> >
> > > [ hadoop]# ls -l
> > /home/hadoop/hadoop-root/mapred/
> > >
> >
> > > total 4
> >
> > >
> >
> > > drwxr-xr-x 2 root root 4096 Nov 26 16:49 local
> >
> > >
> >
> > > [ hadoop]#
> >
> > >
> >
> > > [ hadoop]# ls -l
> > /home/hadoop/hadoop-root/mapred/local/
> > >
> >
> > > total 0
> >
> > > Thanks and Best Regards,
> >
> > >
> >
> > > Krishna Kumar
> >
> > >
> >
> > > Senior Storage Engineer
> >
> > >
> >
> > > Why do we have to die? If we had to die, and
> > everything is gone after
> > > that, then nothing else matters on this earth -
> > everything is
> > > temporary,
> >
> > > at least relative to me.
> >
> > > DISCLAIMER:
> >
> > >
> >
> ---------------------------------------------------------------------------
> > > --------------------------------------------
> >
> > > The contents of this e-mail and any
> > attachment(s) are confidential
> > > and
> >
> > > intended
> >
> > > for the named recipient(s) only.
> >
> > > It shall not attach any liability on the
> > originator or NECHCL or its
> > > affiliates. Any views or opinions presented in
> >
> > > this email are solely those of the author and
> > may not necessarily
> > > reflect the
> >
> > > opinions of NECHCL or its affiliates.
> >
> > > Any form of reproduction, dissemination,
> > copying, disclosure,
> > > modification,
> >
> > > distribution and / or publication of
> >
> > > this message without the prior written consent
> > of the author of this
> > > e-mail is
> >
> > > strictly prohibited. If you have
> >
> > > received this email in error please delete it
> > and notify the sender
> > > immediately. .
> >
> > >
> >
> ---------------------------------------------------------------------------
> > > --------------------------------------------
> >
> > >
> >
> > >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > DISCLAIMER:
> >
> >
> ---------------------------------------------------------------------------
> > --------------------------------------------
> > The contents of this e-mail and any attachment(s) are confidential and
> >
> > intended
> >
> > for the named recipient(s) only.
> >
> > It shall not attach any liability on the originator or NECHCL or its
> >
> > affiliates. Any views or opinions presented in
> >
> > this email are solely those of the author and may not necessarily reflect
> > the
> > opinions of NECHCL or its affiliates.
> >
> > Any form of reproduction, dissemination, copying, disclosure,
> modification,
> >
> > distribution and / or publication of
> >
> > this message without the prior written consent of the author of this
> e-mail
> > is
> > strictly prohibited. If you have
> >
> > received this email in error please delete it and notify the sender
> >
> > immediately. .
> >
> >
> ---------------------------------------------------------------------------
> > --------------------------------------------
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message