Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8C30D18B82 for ; Thu, 15 Oct 2015 06:37:42 +0000 (UTC) Received: (qmail 27937 invoked by uid 500); 15 Oct 2015 06:37:36 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 27848 invoked by uid 500); 15 Oct 2015 06:37:36 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 27837 invoked by uid 99); 15 Oct 2015 06:37:36 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Oct 2015 06:37:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 0136A180A41 for ; Thu, 15 Oct 2015 06:37:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.9 X-Spam-Level: ** X-Spam-Status: No, score=2.9 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 8CBw6kZx8oVO for ; Thu, 15 Oct 2015 06:37:26 +0000 (UTC) Received: from mail-yk0-f181.google.com (mail-yk0-f181.google.com [209.85.160.181]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 57722439CB for ; Thu, 15 Oct 2015 06:37:26 +0000 (UTC) Received: by ykfy204 with SMTP id y204so46389064ykf.1 for ; Wed, 14 Oct 2015 23:37:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=HBEpOZ/EhR8qMhpW6NaKQipdfPE13pHhioMIj7Cn/Ow=; b=SONHzOJKa8wpGFb/r70KG4XoDdqRg/vOn+mI0KJTQonyaN8iD8Tc0dl9NkUmbWodev nbAH7LwdSC7ostPnmX096UnYXYOA6/Kle7Xq1ZSNhIAdrOmra/So+s6dd60sU2EQCfxb E6Q6g1s+LtGe1NdQ45a6Dy5IYo44fvKhPDYrelv7yPE9ocVgnjfWh03jEXlhxkK6LYWn wcLkNpAqUm/M0WCbhPAA1+bEHnGyPRk5ViWWl3SrFog6e+2K+N0dHgAYPG6OmEX89xSJ Ou18zg1xWV+m1f1BOhDKdPvfoh81sfuxeJkXug1JdnME9VcA2//34L8meyT3nnw40EQ+ juuw== X-Received: by 10.129.88.68 with SMTP id m65mr5948660ywb.22.1444891040719; Wed, 14 Oct 2015 23:37:20 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.79.15 with HTTP; Wed, 14 Oct 2015 23:36:51 -0700 (PDT) From: Phil Kallos Date: Wed, 14 Oct 2015 23:36:51 -0700 Message-ID: Subject: Re: Spark 1.5 Streaming and Kinesis To: user@spark.apache.org Content-Type: multipart/alternative; boundary=001a11492bc638ac4b05221ee91c --001a11492bc638ac4b05221ee91c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Not a dumb question, but yes I updated all of the library references to 1.5, including (even tried 1.5.1). // Versions.spark set elsewhere to "1.5.0" "org.apache.spark" %% "spark-streaming-kinesis-asl" % Versions.spark % "provided" I am experiencing the issue in my own spark project, but also when I try to run the spark streaming kinesis example that comes in spark/examples Tried running the streaming job locally, and also in EMR with release 4.1.0 that includes Spark 1.5 Very strange! > ---------- Forwarded message ---------- From: "Jean-Baptiste Onofr=C3=A9" > To: user@spark.apache.org > Cc: > Date: Thu, 15 Oct 2015 08:03:55 +0200 > Subject: Re: Spark 1.5 Streaming and Kinesis > Hi Phil, > KinesisReceiver is part of extra. Just a dumb question: did you update > all, including the Spark Kinesis extra containing the KinesisReceiver ? > I checked on tag v1.5.0, and at line 175 of the KinesisReceiver, we see: > blockIdToSeqNumRanges.clear() > which is a: > private val blockIdToSeqNumRanges =3D new mutable.HashMap[StreamBlockId, > SequenceNumberRanges] > with mutable.SynchronizedMap[StreamBlockId, SequenceNumberRanges] > So, it doesn't look fully correct to me. > Let me investigate a bit this morning. > Regards > JB > On 10/15/2015 07:49 AM, Phil Kallos wrote: > We are trying to migrate from Spark1.4 to Spark1.5 for our Kinesis > streaming applications, to take advantage of the new Kinesis > checkpointing improvements in 1.5. > However after upgrading, we are consistently seeing the following error: > java.lang.ClassCastException: scala.collection.mutable.HashMap cannot be > cast to scala.collection.mutable.SynchronizedMap > at > > org.apache.spark.streaming.kinesis.KinesisReceiver.onStart(KinesisReceive= r.scala:175) > at > > org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(Rece= iverSupervisor.scala:148) > at > > org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupe= rvisor.scala:130) > at > > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpo= int$$anonfun$9.apply(ReceiverTracker.scala:542) > at > > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpo= int$$anonfun$9.apply(ReceiverTracker.scala:532) > at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:198= 4) > at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:198= 4) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:88) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java= :1145) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:615) > at java.lang.Thread.run(Thread.java:745) > I even get this when running the Kinesis examples : > http://spark.apache.org/docs/latest/streaming-kinesis-integration.html > with > bin/run-example streaming.KinesisWordCountASL > Am I doing something incorrect? > > > -- > Jean-Baptiste Onofr=C3=A9 > jbonofre@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com Hi, > --001a11492bc638ac4b05221ee91c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Not a dumb question, but yes I updated all of the lib= rary references to 1.5, including =C2=A0(even tried 1.5.1).

<= /div>
// Versions.spark set elsewhere to "1.5.0"
&q= uot;org.apache.spark" %% "spark-streaming-kinesis-asl" % Ver= sions.spark % "provided"

I am experi= encing the issue in my own spark project, but also when I try to run the sp= ark streaming kinesis example that comes in spark/examples

Tried running the streaming job locally, and also in EMR with rele= ase 4.1.0 that includes Spark 1.5

Very strange!
=C2=A0
-----= ----- Forwarded message ----------=C2=A0
From:=C2=A0"Jean-Baptiste Onofr=C3= =A9" <jb@nanthrax.net>To:=C2=A0user@spark.apache.org
Cc:=C2=A0
Date:=C2=A0T= hu, 15 Oct 2015 08:03:55 +0200
S= ubject:=C2=A0Re: Spark 1.5 Streaming and Kinesis
Hi Phil,
KinesisReceiver is part of extra. Just a dumb questio= n: did you update all, including the Spark Kinesis extra containing the Kin= esisReceiver ?
I checked on tag v1.5.0, and at line 175 of the KinesisReceiver, = we see:
blockIdToSeqNumRanges.clear()
which is a:
private val blockIdToSeqNumRanges =3D ne= w mutable.HashMap[StreamBlockId, SequenceNumberRanges]
=C2=A0 =C2=A0 with mutable.SynchronizedMap[StreamBlockId, SequenceNumberRanges]<= br style=3D"font-size:12.8px">So, it doesn= 't look fully correct to me.
Let me investigate a bit this morning.Regards
JB
On 10/15/2015 07:49 AM, Phil Kallos wro= te:
We are trying to migrate from Spark1.4 to Spark1.5 for our Ki= nesis
streaming applications, to take advantage of the new Kinesis
ch= eckpointing improvements in 1.5.
However after upgrading, we are consist= ently seeing the following error:
java.lang.ClassCastException: scala.co= llection.mutable.HashMap cannot be
cast to scala.collection.mutable.Sync= hronizedMap
at
org.apache.spark.streaming.kinesis.KinesisReceiver.onS= tart(KinesisReceiver.scala:175)
at
org.apache.spark.streaming.receive= r.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:148)
at
o= rg.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervi= sor.scala:130)
at
org.apache.spark.streaming.scheduler.ReceiverTracke= r$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:542)
at=
org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEnd= point$$anonfun$9.apply(ReceiverTracker.scala:532)
at org.apache.spark.Sp= arkContext$$anonfun$37.apply(SparkContext.scala:1984)
at org.apache.spar= k.SparkContext$$anonfun$37.apply(SparkContext.scala:1984)
at org.apache.= spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.sp= ark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Execu= tor$TaskRunner.run(Executor.scala:214)
at
java.util.concurrent.Thread= PoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.con= current.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at ja= va.lang.Thread.run(Thread.java:745)
I even get this when running the Kin= esis examples :
http://spar= k.apache.org/docs/latest/streaming-kinesis-integration.html=C2=A0withbin/run-example streaming.KinesisWordCountASL
Am I doing something inc= orrect?


--=C2=A0
Jean-Baptiste Onofr= =C3=A9
jbonofre@apache.org
http://blog.nanthrax.net
Talen= d -=C2=A0http://www.talend.com
Hi,
--001a11492bc638ac4b05221ee91c--