spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tian Zhang <tzhang...@yahoo.com>
Subject updateStateByKey and stack overflow
Date Sat, 10 Oct 2015 18:37:19 GMT
Hi, I am following the spark streaming stateful application example and write 
a simple counting application with updateStateByKey.
val keyStateStream = actRegBatchCountStream.updateStateByKey(update, new
HashPartitioner(ssc.sparkContext.defaultParallelism), true, initKeyStateRDD)
   
This runs for a few hours and hit the following stack overflow issue.
Any idea?

15/10/10 18:30:08 INFO BlockManagerInfo: Added broadcast_5249_piece0 in
memory on ip-10-112-11-64.ec2.internal:60489 (size: 16.5 KB, free: 4.1 GB)
15/10/10 18:30:08 WARN TaskSetManager: Lost task 4.0 in stage 129045.0 (TID
175432, ip-10-112-11-64.ec2.internal): java.lang.StackOverflowError
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1982)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
	at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
	at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
	at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
	at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
	at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/updateStateByKey-and-stack-overflow-tp25015.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message