hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer <awittena...@linkedin.com>
Subject Re: Too large class path for map reduce jobs
Date Fri, 17 Sep 2010 16:01:56 GMT

On Sep 17, 2010, at 4:56 AM, Henning Blohm wrote:

> When running map reduce tasks in Hadoop I run into classpath issues. Contrary to previous
posts, my problem is not that I am missing classes on the Task's class path (we have a perfect
solution for that) but rather find too many (e.g. ECJ classes or jetty).

The fact that you mention:

> The libs in HADOOP_HOME/lib seem to contain everything needed to run anything in Hadoop
which is, I assume, much more than is needed to run a map reduce task.

hints that your perfect solution is to throw all your custom stuff in lib.  If so, that's
a huge mistake.  Use distributed cache instead.
View raw message