mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pat Ferrel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAHOUT-1464) Cooccurrence Analysis on Spark
Date Thu, 22 May 2014 15:43:07 GMT

    [ https://issues.apache.org/jira/browse/MAHOUT-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14006038#comment-14006038
] 

Pat Ferrel commented on MAHOUT-1464:
------------------------------------

Agreed about LLR. Never saw it underperform the other measures for CF.

I'm using a very small dataset with minSplits = 2 so I can check the actual values. Then running
the epinions, which I can't really check for value. Testing is going through the ItemSimilarityDriver
from MAHOUT-1541 so the write is not using your patch. 

Unfortunately I lied. Works with HDFS in and out but on a multi-threaded local[4] standalone
Spark. Setting to my cluster master still fails. The error message is about connection refused
so there is something is still not configured correctly on my cluster. I had to use fully
qualified URIs to the data because Spark was defaulting to the wrong locations. All pointing
to a bad Spark build or conf. Spark-shell seems to work fine on the cluster. Anyway, I'll
reinstall Spark and try again. Sorry for the false alarm.

> Cooccurrence Analysis on Spark
> ------------------------------
>
>                 Key: MAHOUT-1464
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-1464
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Collaborative Filtering
>         Environment: hadoop, spark
>            Reporter: Pat Ferrel
>            Assignee: Sebastian Schelter
>             Fix For: 1.0
>
>         Attachments: MAHOUT-1464.patch, MAHOUT-1464.patch, MAHOUT-1464.patch, MAHOUT-1464.patch,
MAHOUT-1464.patch, MAHOUT-1464.patch, run-spark-xrsj.sh
>
>
> Create a version of Cooccurrence Analysis (RowSimilarityJob with LLR) that runs on Spark.
This should be compatible with Mahout Spark DRM DSL so a DRM can be used as input. 
> Ideally this would extend to cover MAHOUT-1422. This cross-cooccurrence has several applications
including cross-action recommendations. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message