mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (MAHOUT-1636) Class dependencies for the spark module are put in a job.jar, which is very inefficient
Date Mon, 29 Dec 2014 17:01:13 GMT


ASF GitHub Bot commented on MAHOUT-1636:

GitHub user pferrel opened a pull request:


    Started out simplifying driver code and making changes to all drivers to support that.
Then ran into the fat job.jar issue of MAHOUT-1636 so created a slimmed down version of the
old job.jar by adding excludes to job.xml and changing the name to "dependencies.jar"
    The new jar works for spark-itemsimilarity and spark-row-similarity but needs to be tested
for the naive bayes drivers. 
    The dependencies.jar still contains a lot of stuff from mrlegacy, some is in external
projects, like jackson that can be excluded with this mechanism but also a lot of mahout code
that is unneeded in this jar. This later case would require some other mechanism than a simple
<exclude> clause in the assembly xml file.
    I believe the new dependencies.jar is the only thing that needs to be on the classpath
when running spark drivers or the spark-shell. I haven't changed this but it is a further
refinement we can try.

You can merge this pull request into a Git repository by running:

    $ git pull MAHOUT-1636

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #69
commit d0f64205a116853aa471dd1361a635167da15fcc
Author: pferrel <>
Date:   2014-12-27T23:43:41Z

    simplified driver and made required changes to all, note: left job assembly untouched

commit 5e32612b4ca9400287c43e4130478f04207fbcfa
Author: pferrel <>
Date:   2014-12-29T16:49:10Z

    creating a trimmed down all-deps dependencies.jar for spark drivers


> Class dependencies for the spark module are put in a job.jar, which is very inefficient
> ---------------------------------------------------------------------------------------
>                 Key: MAHOUT-1636
>                 URL:
>             Project: Mahout
>          Issue Type: Bug
>          Components: spark
>    Affects Versions: 1.0-snapshot
>            Reporter: Pat Ferrel
>            Assignee: Ted Dunning
>             Fix For: 1.0-snapshot
> using a maven plugin and an assembly job.xml a job.jar is created with all dependencies
including transitive ones. This job.jar is in mahout/spark/target and is included in the classpath
when a Spark job is run. This allows dependency classes to be found at runtime but the job.jar
include a great deal of things not needed that are duplicates of classes found in the main
mrlegacy job.jar.  If the job.jar is removed, drivers will not find needed classes. A better
way needs to be implemented for including class dependencies.
> I'm not sure what that better way is so am leaving the assembly alone for now. Whoever
picks up this Jira will have to remove it after deciding on a better method.

This message was sent by Atlassian JIRA

View raw message