hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Basnight <mbasni...@gmail.com>
Subject Re: Maven and Mini MR Cluster
Date Thu, 04 Feb 2010 15:35:21 GMT
Ya with the hadoop_home stuff i was grasping at straws. My mini MR Cluster has a valid classpath
i assume, since my entire test runs (thru 3 mapreduce jobs via the localrunner) before it
gets to the mini MR cluster portion. Is it possible to print out the classpath thru the JVMManager
or anything else like that for debugging purposes?

mb

On Feb 4, 2010, at 5:55 AM, Steve Loughran wrote:

> Michael Basnight wrote:
>> Im using maven to run all my unit tests, and i have a unit test that creates a mini
mr cluster. When i create this cluster, i get classdefnotfound errors for the core hadoop
libs (Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Child). When i
run the same test w/o creating the mini cluster, well.. it works fine. My HADOOP_HOME is set
to the same version as my mvn repo, and points to a valid installation of hadoop. When i validate
the classpath thru maven, (dependency:build-classpath), it says that the core libs are on
the classpath as well (sourced from my .m2 repository). I just cant figure out why hadoop's
mini cluster cant find those jars. Running hadoop 0.20.0. Any suggestions?
> 
> the miniMR cluster does everything in memory, and doesnt look at HADOOP_HOME, which is
only for the shell scripts.
> 
> It sounds like you need hadoop-mapreduce on your classpath. Sounds like. the Child class
is the entry point used when creating new JVMs, and it is that classpath that isn't right,
which is a forked JVM from the one the MiniMRCluster was created in.


Mime
View raw message