spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jay vyas <jayunit100.apa...@gmail.com>
Subject Is there a way for scala compiler to catch unserializable app code?
Date Mon, 17 Nov 2014 00:12:21 GMT
This is more a curiosity than an immediate problem.

Here is my question: I ran into this easily solved issue
http://stackoverflow.com/questions/22592811/task-not-serializable-java-io-notserializableexception-when-calling-function-ou
recently.  The solution was to replace my "class" with a scala singleton,
which i guess is readily serializable.

So its clear that spark needs to serialize objects which carry the driver
methods for an app, in order to run... but I'm wondering,,, maybe there is
a way to change or update the spark API to catch unserializable spark apps
at compile time?


-- 
jay vyas

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message