hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joey Echeverria <j...@cloudera.com>
Subject Re: Adding mahout math jar to hadoop mapreduce execution
Date Wed, 01 Feb 2012 13:50:32 GMT
The -libjars feature is needed if you use the classes in your remote code (map and reduce functions).
Is it possible you only use it in your main() method?

-Joey

Sent from my iPhone

On Feb 1, 2012, at 3:38, Daniel Quach <danquach@cs.ucla.edu> wrote:

> Thanks, this seems to make it work. In fact, I did not even need to specify lib jars
in the command line…should I be worried that it doesn't work that way?
> 
> On Jan 31, 2012, at 4:09 PM, Joey Echeverria wrote:
> 
>> You also need to add the jar to the classpath so it's available in
>> your main. You can do soemthing like this:
>> 
>> HADOOP_CLASSPATH=/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar
>> hadoop jar ...
>> 
>> -Joey
>> 
>> On Tue, Jan 31, 2012 at 1:38 PM, Daniel Quach <danquach@cs.ucla.edu> wrote:
>>> For Hadoop 0.20.203 (the latest stable), is it sufficient to do this to parse
the lib jars from the command line?
>>> 
>>> public static main (String args[]) {
>>> Configuration conf = new Configuration();
>>> String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
>>> 
>>> Job job = new Job(conf, "makevector");
>>> job.setJarByClass(MakeVector.class);
>>> // etc other code for mappers/reducers
>>> }
>>> 
>>> I'm thinking I am missing a step here which is why it won't load the mahout jar
while running the map reduce.
>>> 
>>> On Jan 30, 2012, at 11:10 PM, Daniel Quach wrote:
>>> 
>>>> I compiled using javac:
>>>> 
>>>> javac -classpath :/usr/local/hadoop/hadoop-core-0.20.203.0.jar:/usr/local/hadoop/lib/commons-cli-1.2.jar:/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar
-d makevector_classes/ MakeVector.java;
>>>> 
>>>> If I don't include the mahout-math jar, it gives me a compile error because
of DenseVector.
>>>> 
>>>> 
>>>> On Jan 30, 2012, at 10:42 PM, Prashant Kommireddi wrote:
>>>> 
>>>>> How are you building the mapreduce jar? Try not to include the Mahout
dist
>>>>> while building MR jar, and include it only on "-libjars" option.
>>>>> 
>>>>> On Mon, Jan 30, 2012 at 10:33 PM, Daniel Quach <danquach@cs.ucla.edu>
wrote:
>>>>> 
>>>>>> I have been compiling my mapreduce with the jars in the classpath,
and I
>>>>>> believe I need to also add the jars as an option to -libjars to hadoop.
>>>>>> However, even when I do this, I still get an error complaining about
>>>>>> missing classes at runtime. (Compilation works fine).
>>>>>> 
>>>>>> Here is my command:
>>>>>> hadoop jar makevector.jar org.myorg.MakeVector -libjars
>>>>>> /usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar input/
output/
>>>>>> 
>>>>>> This is the error I receive:
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>> org/apache/mahout/math/DenseVector
>>>>>> 
>>>>>> I wonder if I am using the GenericOptionsParser incorrectly? I'm
not sure
>>>>>> if there is a deeper problem here.
>>>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> -- 
>> Joseph Echeverria
>> Cloudera, Inc.
>> 443.305.9434
> 

Mime
View raw message