mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pat Ferrel <>
Subject Re: cf/couccurence code
Date Mon, 07 Jul 2014 16:18:58 GMT
OK, I’m spending more time on this than I have to spare. The test class extends MahoutLocalContext,
which provides an implicit Spark context. I haven’t found a way to test parallel execution
of cooccurrence without it. So far the only obvious option is to put cf into math-scala but
the tests would have to remain in spark and that seems like trouble so I’d rather not do

I suspect as more math-scala consuming algos get implemented this issue will proliferate.
We will have implementations that do not require Spark but tests that do. We could create
a new sub-project that allows for this I suppose but a new sub-project will require changes
to SparkEngine and mahout’s script.

If someone (Anand?) wants to offer a PR with some way around this I’d be happy to integrate.

On Jun 30, 2014, at 5:39 PM, Pat Ferrel <> wrote:

No argument, just trying to decide whether to create core-scala or keep dumping anything not
Spark dependent in math-scala. 

On Jun 30, 2014, at 9:32 AM, Ted Dunning <> wrote:

On Mon, Jun 30, 2014 at 8:36 AM, Pat Ferrel <> wrote:

> Speaking for Sebastian and Dmitriy (with some ignorance) I think the idea
> was to isolate things with Spark dependencies something like we did before
> with Hadoop.

Go ahead and speak for me as well here!

I think isolating the dependencies is crucial for platform nimbleness

View raw message