spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Pascal Billaud>
Subject Task not Serializable: Graph is unexpectedly null when DStream is being serialized
Date Mon, 20 Apr 2015 17:44:16 GMT

I am getting this serialization exception and I am not too sure what "Graph
is unexpectedly null when DStream is being serialized" means?

15/04/20 06:12:38 INFO yarn.ApplicationMaster: Final app status: FAILED,
exitCode: 15, (reason: User class threw exception: Task not serializable)
Exception in thread "Driver" org.apache.spark.SparkException: Task not
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(
        at org.apache.spark.util.ClosureCleaner$.clean(
        at org.apache.spark.SparkContext.clean(SparkContext.scala:1435)
Caused by: Graph is unexpectedly null
when DStream is being serialized.
        at org.apache.spark.streaming.dstream.DStream$anonfun$
        at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:985)
        at org.apache.spark.streaming.dstream.DStream.writeObject(

The operation comes down to something like this: => {
val w = StreamState.fetch[K,W](state.prefixKey, tuple._1)
(tuple._1, (tuple._2, w)) })

And StreamState being a very simple standalone object:

object StreamState {
  def fetch[K : ClassTag : Ordering, V : ClassTag](prefixKey: String, key:
K) : Option[V] = None

However if I remove the context bounds from K in fetch e.g. removing
ClassTag and Ordering then everything is fine.

If anyone has some pointers, I'd really appreciate it.


View raw message