mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From PierLorenzo Bianchini <>
Subject fast performance way of writing preferences to file?
Date Fri, 03 Apr 2015 10:26:47 GMT
Hello everyone,
I'm new to mahout, to recommender systems and to the mailing list.

I''m trying to find a (fast) way to write back preferences to a file. I tried a few methods
but I'm sure there must be a better approach.
Here's the deal (you can find the same post in stackoverflow[1]).
I have a training dataset of 800.000 records from 6000 users rating 3900 movies. These are
stored in a comma separated file like: userId,movieId,preference. I have another dataset (200.000
records) in the format: userId,movieId. My goal is to use the first dataset as a training-set,
in order to determine the missing preferences of the second set.

So far, I managed to load the training dataset and I generated user-based recommendations.
This is pretty smooth and doesn't take too much time. But I'm struggling when it comes to
writing back the recommendations.

The first method I tried is:

 * read a line from the file and get the userId,movieId tuple.
 * retrieve the calculated preference with estimatePreference(userId, movieId)
 * append the preference to the line and save it in a new file
This works, but it's incredibly slow (I added a counter to print every 10.000th iteration:
after a couple of minutes it had only printed once. I have 8GB-RAM with an i7-core... how
long can it take to process 200.000 lines?!)

My second choise was:

 * create a new FileDataModel with the second dataset
 * do something like this: newDataModel.setPreference(userId, movieId, recommender.estimatePreference(userId,

Here I get several problems:
 * at runtime: java.lang.UnsupportedOperationException (as I found out in [2], FileDataModel
actually can't be updated. I don't understand why the function setPreference exists in the
first place...)
 * The API of FileDataModel#setPreference states "This method should also be considered relatively

I read around that a solution would be to use delta files, but I couldn't find out what that
actually means. Any suggestion on how I could speed up my writing-the-preferences process?
Thank you!

Pier Lorenzo


View raw message