mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel McEnnis <dmcen...@gmail.com>
Subject Re: https://issues.apache.org/jira/browse/MAHOUT-662 -- misplaced calls to setJarByClass
Date Fri, 08 Apr 2011 18:24:18 GMT
A quick mention:

I'm tracking down a rogue dependency on a run-time JSONException
class-not-found.  In the process, I discovered that Hadoop has two
different sets of map reduce classes.  They have similar but not
exactly the same syntax.  One is used in the Cloudera tutorial for
Hadoop.  The other in the Mahout code.  (I can provide more details if
needed.) There may eventually be differing dependencies between these
two sets of code in Hadoop further complicating the matter.

Daniel.

On Fri, Apr 8, 2011 at 2:02 PM, Sean Owen <srowen@gmail.com> wrote:
> This I suppose is another reason to repackage it all into one library.
>
> I had thought the solution is to reference a class which must exist in the
> top-level jar (I pick the Mapper or Reducer usually). That stuff will be in
> a classloader that sees everything. Or so I would have guessed.
>
> From what I make out your test may suggest otherwise. It seems to suggest
> there's not a good way to deal with multiple jars even when rolled into one.
> That is to say pick a class in another jar in this assemblage and you'd get
> a different error. Right?
>
> So repackage?
>
> On Fri, Apr 8, 2011 at 6:19 PM, Benson Margulies <bimargulies@gmail.com>wrote:
>
>> I just tried following my own prescription in the above-captioned
>> jira, and it doesn't work. ClassNotFound on
>> org.apache.mahout.math.VectorWritable.
>>
>> That class is in mahout-utils, and the mahout-utils jar is sitting
>> there in lib/ of the example job jar.
>>
>> This morning, I built a minimal test case of the hadoop mechanism at
>> work here. I put one very skinny class in the ordinary part of the
>> jar, and all the other code in other jars in the lib directory of the
>> jar. I even included loading a class via the thread context
>> classloader.
>>
>> It all worked perfectly fine.
>>
>> So, the nonfunction of this with Mahout has something to do with ...
>> Mahout.
>>
>> And I have an idea of what. When we call:
>>
>> job.setJarByClass(ThreadContextClasspathTestJob.class); (for the
>> appropriate class of ours).
>>
>> the class we pass in is the driver class that live in mahout-core, NOT
>> a class from the example job jar.
>>
>> I think this is wrong and explains the malfunction. The question is,
>> how to fix it?
>>
>

Mime
View raw message