mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pat Ferrel <...@occamsmachete.com>
Subject Re: ClassNotFoundException while using RecommenderJob
Date Fri, 16 Mar 2012 22:28:46 GMT
Thanks, [1] is a very good reference for ClassNotFoundException.

As you say in [2] the mahout script does not accept hadoop job 
parameters in all cases although I hope it does in the future, 
especially where a parameter to the job is a classname (seq2sparse for 
instance).

What I had to do was copy my custom analyzer jar into $HADOOP_HOME/lib 
on the master node. Evidently a symlink does not work, it appears you 
have to copy each jar you want to the directory.

Then don't forget to stop and start hadoop because as [1] says it 
packages the libs at startup.

On 3/16/12 7:12 AM, Ioan Eugen Stan wrote:
> Pe 15.03.2012 15:51, Pat Ferrel a scris:
>> Can you elaborate on how to set these classpaths?
>>
>> The job takes a classpath param, (from mahout --help)
>>
>> -libjars <paths> comma separated jar files to include in
>> the classpath.
>>
>> is the needed code bundled with the job or does the path need to be
>> available to hadoop/mahout on all nodes? Also is this the only thing
>> that affects the classpath to any code outside the jar file?
>>
>> On 3/15/12 5:33 AM, Ioan Eugen Stan wrote:
>>> Also, if you submit a job there are two class paths that you must be
>>> aware of:
>>> - the one that starts the job submitting app
>>> - the one that will be available to the job on each node
>>
>
> Hi Pat,
>
> To set up your main app class-path use HADOOP_CLASS_PATH env var. For 
> the Job class-path it's best explained here [1], although, from my 
> experience -libjars did not work as advertised. It's hadoop version 
> dependant probably but I may be wrong. Test and see for yourself.
>
> Also see my post on common-user about this issue [2]. Search for 
> ClassNotFoundException: libjars .
>
>
> [1] 
> http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
> [2] http://hadoop.markmail.org/thread/st3cy7alcodlztng
>
> Hope this helps,
>

Mime
View raw message