spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin Woodbury <coli...@gmail.com>
Subject MLlib ALS MatrixFactorizationModel.save fails consistently
Date Thu, 07 Apr 2016 22:56:50 GMT
Hi all,

I've implemented most of a content recommendation system for a client.
However, whenever I attempt to save a MatrixFactorizationModel I've
trained, I see one of four outcomes:

1. Despite "save" being wrapped in a "try" block, I see a massive stack
trace quoting some java.io classes. The Model isn't written.
2. Same as the above, but the Model *is* written. It's unusable however, as
it's missing many of the files it should have, particularly in the
"product" folder.
3. Same as the above, but sbt crashes completely.
4. No massive stack trace, and the Model seems to be written. Upon being
loaded by another Spark context and fed a user ID, it claims the user isn't
present in the Model.

Case 4 is pretty rare. I see these failures both locally and when I test on
a Google Cloud instance with much better resources.

Note that `ALS.trainImplicit` and `model.save` are being called from within
a Future. Could it be possible that Play threads are closing before Spark
can finish, thus interrupting it somehow?

We are running Spark 1.6.1 within Play 2.4 and Scala 2.11. All these
failures have occurred while in Play's Dev mode in SBT.

Thanks for any insight you can give.

Mime
View raw message