Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BE11F184DA for ; Tue, 9 Jun 2015 02:24:09 +0000 (UTC) Received: (qmail 46957 invoked by uid 500); 9 Jun 2015 02:24:07 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 46878 invoked by uid 500); 9 Jun 2015 02:24:06 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 46868 invoked by uid 99); 9 Jun 2015 02:24:06 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jun 2015 02:24:06 +0000 X-ASF-Spam-Status: No, hits=3.7 required=5.0 tests=FORGED_YAHOO_RCVD,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLYTO_END_DIGIT,HTML_FONT_LOW_CONTRAST,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of m_albert137@yahoo.com designates 98.138.91.178 as permitted sender) Received: from [98.138.91.178] (HELO nm18-vm4.bullet.mail.ne1.yahoo.com) (98.138.91.178) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jun 2015 02:21:51 +0000 Received: from [98.138.100.102] by nm18.bullet.mail.ne1.yahoo.com with NNFMP; 09 Jun 2015 02:23:38 -0000 Received: from [98.138.226.166] by tm101.bullet.mail.ne1.yahoo.com with NNFMP; 09 Jun 2015 02:23:38 -0000 Received: from [127.0.0.1] by omp1067.mail.ne1.yahoo.com with NNFMP; 09 Jun 2015 02:23:38 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 268845.71939.bm@omp1067.mail.ne1.yahoo.com X-YMail-OSG: OzJ7.mwVM1lo0CeligoL2uvHf_Lo6SWt7_n2MbPVsDr.P20qrJWsFM05T97S3L5 AZQcex8mDQ5HI_NR2uGvG0IGnnTSX2RBFZw8bcQL6gDpgo9bBkIOyXwPmvLfPpXKsR8Bzr9QCr6b zYcugvcKypz_1SJw1eSaX4clFo_GnceGnRBAj77k_mFGHC31.YNNJEfTdkN4al781tEm12y6H2wJ vDaAEHpPO.01k5thEMua8T8VCUyR_QMg86Vn1prrdXhqKCOZc2gv8qK7J2nZmeGbSj4dIzVp6Pm3 LRLzhCAW8IMmK4n7_66b10p10yqzgkiOxaF0yKjDeyrdqfOogPOFgM2PvtK4ElUA74awBECap7IM IeQVyeFkykkUmb68I_gOfLW3hRen_WjvtyHwEQ3ZBCF42wc0Ve7Es22pw2e36v8HXfnRYnrDFNl2 FujS1ZxBGOjAM1KZNP2YbCjVyp8g4HNkPgJv5bSfuL7kRBFt5Sc7FEDomcXbIRyJET4ACQNbZ9BJ JtRsnpQ4k4FvA Received: by 98.138.101.160; Tue, 09 Jun 2015 02:23:37 +0000 Date: Tue, 9 Jun 2015 02:23:37 +0000 (UTC) From: Michael Albert Reply-To: Michael Albert To: "bit1129@163.com" , user Message-ID: <962195262.7639778.1433816617393.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: <201506091001412781300@163.com> References: <201506091001412781300@163.com> Subject: Re: Wired Problem: Task not serializable[Spark Streaming] MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_7639777_635669573.1433816617383" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_7639777_635669573.1433816617383 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Note that in scala, "return" is a non-local return:=C2=A0https://tpolecat.g= ithub.io/2014/05/09/return.htmlSo that "return" is *NOT* returning from the= anonymous function, but attempting to return from the enclosing method, i.= e., "main".Which is running on the driver, not on the workers.So on the wor= kers, there is no where to which the "return" can jump.Hence it is not seri= alizable. Good luck.-Mike From: "bit1129@163.com" To: user =20 Sent: Monday, June 8, 2015 10:01 PM Subject: Re: Wired Problem: Task not serializable[Spark Streaming] =20 #yiv1823860044 body {line-height:1.5;}#yiv1823860044 blockquote {margin-top= :0px;margin-bottom:0px;margin-left:0.5em;}#yiv1823860044 div.yiv1823860044f= oxdiv20150609100051035499 {}#yiv1823860044 body {font-size:10.5pt;color:rgb= (0, 0, 0);line-height:1.5;}Could someone help explain what happens that lea= ds to the Task not serializable issue?Thanks. bit1129@163.com =C2=A0From:=C2=A0bit1129@163.comDate:=C2=A02015-06-08=C2=A019:08To:=C2=A0us= erSubject:=C2=A0Wired Problem: Task not serializable[Spark Streaming]Hi,=C2= =A0With the following simple code, I got an exception that complains=C2=A0T= ask not serializable.=C2=A0The root cause is I use return in map foreach lo= op Why "return in map foreach loop" cause the=C2=A0Task not serializable probl= em, can someone please this to me? import org.apache.spark.SparkConf import org.apache.spark.streaming._ import scala.collection.mutable object NetCatStreamingWordCount3 { def main(args: Array[String]) { val conf =3D new SparkConf().setAppName("NetCatWordCount") conf.setMaster("local[3]") val ssc =3D new StreamingContext(conf, Seconds(5)) val lines =3D ssc.socketTextStream("localhost", 9999) lines.foreachRDD(rdd =3D> { rdd.foreachPartition(partitionIterable=3D> { val map =3D mutable.Map[String, String]() while(partitionIterable.hasNext) { val v =3D partitionIterable.next() map +=3D v ->v } map.foreach(entry =3D> { if (entry._1.equals("abc")) { return; =C2=A0//This is the root cause that cause the=C2=A0=C2=A0Task not = serializable. } }) }) }) ssc.start() ssc.awaitTermination() } }bit1129@163.com ------=_Part_7639777_635669573.1433816617383 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Note that in scala, "return" is a n= on-local return: https://tpolecat.github.io/= 2014/05/09/return.html
So that "return" is *NOT* returning from the anonymous function, bu= t attempting to return from the enclosing method, i.e., "main".
Which is running on the driver, not o= n the workers.
So on the w= orkers, there is no where to which the "return" can jump.
Hence it is not serializable.

Good luck.
-Mike


From: "bit1129@163.com" <bit1129@163.com>
To: user <user@spark.apache.org>
<= span style=3D"font-weight: bold;">Sent: Monday, June 8, 2015 10:= 01 PM
Subject: Re: Wir= ed Problem: Task not serializable[Spark Streaming]

Could someone help= explain what happens that leads to the Task not serializable issue?
<= div>Thanks.

<= div>bit1129@163.com
 
Date: 2015-06-08 19:08
To: user
Subject: Wired Problem: Task not seriali= zable[Spark Streaming]
Hi, 
With the following simple code, I = got an exception that complains Task not serializable= . 
The root cause is I use return in map foreach loop=

Why "return in map foreach loop" cause the Task not serializable problem, can someone please this to me?



import org.apache.spark.SparkConf
import org.apache.spark.streaming._

import scala.collection.mutable

object NetCatStreamingWordCount3 {
def main(args: Array[String]) {
val conf =3D new SparkConf().setAppName("NetCatWordCount")
conf.setMaster("local[3]")
val ssc =3D new StreamingContext(conf, Seconds(5))
val lines =3D ssc.socketTextStream("localhost", 9999)
lines.foreachRDD(rdd =3D> {
rdd.foreachPartition(partitionIterable=3D> {
val map =3D mutable.Map[String, String]()
while(partitionIterable.hasNext) {
val v =3D partitionIterable.next()
map +=3D v ->v
}

map.foreach(entry =3D> {
if (entry._1.equals("abc")) {
return;  //This is the root cause that cause the =  Task not serializable.
}
})

})
})
ssc.start()
ssc.awaitTermination()
}
}
<= div>bit1129@163.com


------=_Part_7639777_635669573.1433816617383--