hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jon Blower" <...@mail.nerc-essc.ac.uk>
Subject RE: Running WordCount in pseudo-distributed configuration
Date Wed, 01 Mar 2006 08:15:14 GMT
Hi Doug,

It worked - but only after I changed the patch slightly.  The
HADOOP_HOME/webapps directory must come *before* the jar files in the
CLASSPATH, i.e:

# for releases, add hadoop jars & webapps to CLASSPATH
if [ -d "$HADOOP_HOME/webapps" ]; then
  CLASSPATH=${CLASSPATH}:$HADOOP_HOME
fi
for f in $HADOOP_HOME/hadoop-*.jar; do
  CLASSPATH=${CLASSPATH}:$f;
done

Now it works fine.  If the HADOOP_HOME/webapps directory comes after the jar
files in the CLASSPATH, the system will look inside the JAR file, find the
webapps application, then I get the same error as before in which it looks
for webapps in a bizarre path:

/users/resc/programs/hadoop-nightly/file:/users/resc/programs/hadoop-nightly
/hadoop-nightly.jar!/webapps

I presume this should be
file:/users/resc/programs/hadoop-nightly/hadoop-nightly.jar!/webapps


In summary, to get an out-of-the-box installation working I had to:

1) Added the HADOOP_HOME/webapps directory to the CLASSPATH, before the JAR
files as above (by editing bin/hadoop)
2) Edited bin/slaves.sh: removed "-o ConnectTimeout=1" from call to ssh
(near the bottom of the file) as this option isn't supported by my version
of ssh
3) setenv HADOOP_OPTS -Dorg.mortbay.xml.XmlParser.NotValidating=true
4) bin/start-all.sh

I have only tried these changes on Red Hat Linux 9 but I'd be optimistic
that they would work on other systems.  It raises the question - does the
webapps file need to be in the JAR file at all?  I'm sure that if it were
removed from the JAR it would still work so long as there was a
HADOOP_HOME/webapps directory to find.

Best regards and thanks for all your help,
Jon

--------------------------------------------------------------
Dr Jon Blower              Tel: +44 118 378 5213 (direct line)
Technical Director         Tel: +44 118 378 8741 (ESSC)
Reading e-Science Centre   Fax: +44 118 378 6413
ESSC                       Email: jdb@mail.nerc-essc.ac.uk
University of Reading
3 Earley Gate
Reading RG6 6AL, UK
--------------------------------------------------------------  

> -----Original Message-----
> From: Doug Cutting [mailto:cutting@apache.org] 
> Sent: 28 February 2006 23:02
> To: hadoop-user@lucene.apache.org
> Subject: Re: Running WordCount in pseudo-distributed configuration
> 
> Jon Blower wrote:
> > I then examined the jobtracker log file 
> > (logs/hadoop-resc-jobtracker-myserver.log).  This contains the (I 
> > think) tell-tale lines:
> > 
> > 060228 220459 Version Jetty/5.1.4
> > 060228 220459 Checking Resource aliases
> > 060228 220459 Web application not found 
> > 
> /users/resc/programs/hadoop-nightly/file:/users/resc/programs/hadoop-n
> > ightly
> > /hadoop-nightly.jar!/webapps
> 
> Ah, I think this is a CLASSPATH problem.
> 
> Try applying the attached patch & tell me if it fixes things.
> 
> Thanks,
> 
> Doug
> 


Mime
View raw message