hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dmaniar <dharin.man...@gmail.com>
Subject Hadoopp_ClassPath issue.
Date Wed, 11 Apr 2012 04:09:10 GMT


I am new to hadoop and its not very familiar with internal working. I had
some questions about HADOOP_CLASSPATH. 

We are currently suppose to use a Hadoop cluster with 4 machines and its
HADOOP_CLASSPATH in hadoop-env.sh is as below.

Now my,
/home/user/app/www/WEB-INF/classes has a class called Application.class

>From a remote machine I submit a map-reduce job to this cluster, with a jar
called MyJar.jar. [This has a Application.class too, but with some

When the TaskTracker spawns a child Java process for the Mapper the
classpath I see is as below in that order,

Lets say my hadoop is installed at: /home/user/hadoop/
[note:- basically this has the modified class that I need to use for my
Map-Reduce job]

Well its clear from this classpath that i will end up using the
Application.class from the classes folder. with gives me incorrect results.

Now my Question is, how do I make sure i reverse the order of 1 & 2.

Some pointer that I found was,
1) if MyJar.jar is not changing much then I can put in a shared location and
modify my hadoop-env.sh to

2) get rid of /home/user/app/www/WEB-INF/classes, from my hadoop-env.sh

3) is there any property taht suggest to add before classpath ?

Any help is greatly appreciated.

To Summarize,
If I have HADOOP_CLASSPTH in hadoop-env.sh already set, then how do I add
application jar before this classpath.

Again. I saw the DistributedCache.java [hadoop src] and the code looks like.

public static void addFileToClassPath(Path file, Configuration conf)
	throws IOException {
	String classpath = conf.get("mapred.job.classpath.files");
	conf.set("mapred.job.classpath.files", classpath == null ? file
	.toString() : classpath + System.getProperty("path.separator")
	+ file.toString());

basically new files are added to the end of existing classpath.


View this message in context: http://old.nabble.com/Hadoopp_ClassPath-issue.-tp33666009p33666009.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

View raw message