Return-Path: 
 <user-return-44207-apmail-spark-user-archive=spark.apache.org@spark.apache.org>
X-Original-To: apmail-spark-user-archive@minotaur.apache.org
Delivered-To: apmail-spark-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D352718136
	for <apmail-spark-user-archive@minotaur.apache.org>;
 Tue, 13 Oct 2015 21:47:32 +0000 (UTC)
Received: (qmail 39166 invoked by uid 500); 13 Oct 2015 21:47:29 -0000
Delivered-To: apmail-spark-user-archive@spark.apache.org
Received: (qmail 39071 invoked by uid 500); 13 Oct 2015 21:47:29 -0000
Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@spark.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@spark.apache.org>
List-Post: <mailto:user@spark.apache.org>
List-Id: <user.spark.apache.org>
Delivered-To: mailing list user@spark.apache.org
Received: (qmail 39061 invoked by uid 99); 13 Oct 2015 21:47:29 -0000
Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2015 21:47:29 +0000
Received: from localhost (localhost [127.0.0.1])
	by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org)
 with ESMTP id E3423C3BB5
	for <user@spark.apache.org>; Tue, 13 Oct 2015 21:47:28 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: 3.15
X-Spam-Level: ***
X-Spam-Status: No, score=3.15 tagged_above=-999 required=6.31
	tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,
	FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=3, SPF_PASS=-0.001,
	URIBL_BLOCKED=0.001] autolearn=disabled
Authentication-Results: spamd1-us-west.apache.org (amavisd-new);
	dkim=pass (2048-bit key) header.d=gmail.com
Received: from mx1-us-east.apache.org ([10.40.0.8])
	by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new,
 port 10024)
	with ESMTP id 34BaFFUA_-N3 for <user@spark.apache.org>;
	Tue, 13 Oct 2015 21:47:21 +0000 (UTC)
Received: from mail-lf0-f42.google.com (mail-lf0-f42.google.com
 [209.85.215.42])
	by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with
 ESMTPS id DC4F4439C4
	for <user@spark.apache.org>; Tue, 13 Oct 2015 21:47:20 +0000 (UTC)
Received: by lfaz124 with SMTP id z124so2872812lfa.1
        for <user@spark.apache.org>; Tue, 13 Oct 2015 14:47:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:date:message-id:subject:from:to:content-type;
        bh=ww1NC+ZY8iO7GsG206u5HhbcDGGXE1G0IkJ7vScKnvA=;
        b=hUrMQd+1F8W1fLzxKuOGTdxZmczSRdFr6cK4zpbTpd9fecBf25kdPnch+Qw+qTcmeN
         V/43Qll9uuGlUfk7icE7iXNfJ6scD0ToquKDodMeGyZP4dWbyRjUSJTYvKBsFqPDtqkq
         VjMOlyxvBy07EAsalLcaYONhXaPOjRuwvjZ6x9T2MNIsT70WP2AtdPtmzgiUH4mLfk+Y
         AGMOPBP8Vv2cp0Fmfh7r2yXtxmM2CCpQP13GwnWZpmRC4kp5vPm0xrqM3Zx60zLTE9hz
         6Po9LMSh7wXWTjL+3rb/H5+NrTbuvFkT97xLTg6biaXn19S7guszl/gh/I4tI1kMItj8
         YABA==
MIME-Version: 1.0
X-Received: by 10.25.24.195 with SMTP id 64mr420391lfy.71.1444772833841; Tue,
 13 Oct 2015 14:47:13 -0700 (PDT)
Received: by 10.25.40.84 with HTTP; Tue, 13 Oct 2015 14:47:13 -0700 (PDT)
Date: Tue, 13 Oct 2015 14:47:13 -0700
Message-ID: 
 <CANu_wT4uLy6VDjFks1r+nubdOvZtieFPaqG97p4iaDt1g=qwBw@mail.gmail.com>
Subject: Spark 1.5 java.net.ConnectException: Connection refused
From: Spark Newbie <sparknewbie1234@gmail.com>
To: user@spark.apache.org
Content-Type: multipart/alternative; boundary=001a11406d388ad6f705220363ad

--001a11406d388ad6f705220363ad
Content-Type: text/plain; charset=UTF-8

Hi Spark users,

I'm seeing the below exception in my spark streaming application. It
happens in the first stage where the kinesis receivers receive records and
perform a flatMap operation on the unioned Dstream. A coalesce step also
happens as a part of that stage for optimizing the performance.

This is happening on my spark 1.5 instance using kinesis-asl-1.5. When I
look at the executor logs I do not see any exceptions indicating the root
cause of why there is no connectivity on xxx.xx.xx.xxx:36684 or when did
that service go down.

Any help debugging this problem will be helpful.

15/10/13 16:36:07 ERROR shuffle.RetryingBlockFetcher: Exception while
beginning fetch of 1 outstanding blocks
java.io.IOException: Failed to connect to
ip-xxx-xx-xx-xxx.ec2.internal/xxx.xx.xx.xxx:36684
        at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:193)
        at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:156)
        at
org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:88)
        at
org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
        at
org.apache.spark.network.shuffle.RetryingBlockFetcher.start(RetryingBlockFetcher.java:120)
        at
org.apache.spark.network.netty.NettyBlockTransferService.fetchBlocks(NettyBlockTransferService.scala:97)
        at
org.apache.spark.network.BlockTransferService.fetchBlockSync(BlockTransferService.scala:89)
        at
org.apache.spark.storage.BlockManager$$anonfun$doGetRemote$2.apply(BlockManager.scala:595)
        at
org.apache.spark.storage.BlockManager$$anonfun$doGetRemote$2.apply(BlockManager.scala:593)
        at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at
org.apache.spark.storage.BlockManager.doGetRemote(BlockManager.scala:593)
        at
org.apache.spark.storage.BlockManager.getRemote(BlockManager.scala:579)
        at org.apache.spark.storage.BlockManager.get(BlockManager.scala:623)
        at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:44)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:262)
        at
org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:139)
        at
org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:135)
        at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
        at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:135)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
        at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
        at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
        at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:69)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:262)
        at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
        at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
        at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
        at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        at org.apache.spark.scheduler.Task.run(Task.scala:88)
        at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused:
ip-xxx-xx-xx-xxx.ec2.internal/xxx.xx.xx.xxx:36684

Thanks,
Bharath

--001a11406d388ad6f705220363ad
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div>Hi Spark users,<br><br></div>I&#39;m seeing=
 the below exception in my spark streaming application. It happens in the f=
irst stage where the kinesis receivers receive records and perform a flatMa=
p operation on the unioned Dstream. A coalesce step also happens as a part =
of that stage for optimizing the performance. <br><br></div>This is happeni=
ng on my spark 1.5 instance using kinesis-asl-1.5. When I look at the execu=
tor logs I do not see any exceptions indicating the root cause of why there=
 is no connectivity on xxx.xx.xx.xxx:36684 or when did that service go down=
. <br><br></div>Any help debugging this problem will be helpful.<br><div><d=
iv><div><div><br>15/10/13 16:36:07 ERROR shuffle.RetryingBlockFetcher: Exce=
ption while beginning fetch of 1 outstanding blocks<br>java.io.IOException:=
 Failed to connect to ip-xxx-xx-xx-xxx.ec2.internal/xxx.xx.xx.xxx:36684<br>=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.network.clie=
nt.TransportClientFactory.createClient(TransportClientFactory.java:193)<br>=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.network.clie=
nt.TransportClientFactory.createClient(TransportClientFactory.java:156)<br>=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.network.nett=
y.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferServic=
e.scala:88)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spa=
rk.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFe=
tcher.java:140)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache=
.spark.network.shuffle.RetryingBlockFetcher.start(RetryingBlockFetcher.java=
:120)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.net=
work.netty.NettyBlockTransferService.fetchBlocks(NettyBlockTransferService.=
scala:97)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark=
.network.BlockTransferService.fetchBlockSync(BlockTransferService.scala:89)=
<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.storage.=
BlockManager$$anonfun$doGetRemote$2.apply(BlockManager.scala:595)<br>=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.storage.BlockManag=
er$$anonfun$doGetRemote$2.apply(BlockManager.scala:593)<br>=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 at scala.collection.mutable.ResizableArray$clas=
s.foreach(ResizableArray.scala:59)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47=
)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.storage=
.BlockManager.doGetRemote(BlockManager.scala:593)<br>=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.storage.BlockManager.getRemote(Bl=
ockManager.scala:579)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.=
apache.spark.storage.BlockManager.get(BlockManager.scala:623)<br>=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.CacheManager.getOrCom=
pute(CacheManager.scala:44)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 a=
t org.apache.spark.rdd.RDD.iterator(RDD.scala:262)<br>=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute=
$2.apply(CoGroupedRDD.scala:139)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGrouped=
RDD.scala:135)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at scala.colle=
ction.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.s=
cala:772)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at scala.collection=
.immutable.List.foreach(List.scala:318)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 at scala.collection.TraversableLike$WithFilter.foreach(Travers=
ableLike.scala:771)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.ap=
ache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:135)<br>=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.rdd.RDD.computeOrReadChe=
ckpoint(RDD.scala:297)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org=
.apache.spark.rdd.RDD.iterator(RDD.scala:264)<br>=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPart=
itionsRDD.scala:38)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.ap=
ache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)<br>=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.rdd.RDD.iterator(RDD.sca=
la:264)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.r=
dd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)<br>=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.rdd.RDD.computeOrReadCheckp=
oint(RDD.scala:297)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.ap=
ache.spark.CacheManager.getOrCompute(CacheManager.scala:69)<br>=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.rdd.RDD.iterator(RDD.sca=
la:262)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.r=
dd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)<br>=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.rdd.RDD.computeOrReadCheckp=
oint(RDD.scala:297)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.ap=
ache.spark.rdd.RDD.iterator(RDD.scala:264)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartiti=
onsRDD.scala:38)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apach=
e.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)<br>=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.rdd.RDD.iterator(RDD.scala:=
264)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.sche=
duler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)<br>=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.scheduler.ShuffleMapTask.runTa=
sk(ShuffleMapTask.scala:41)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 a=
t org.apache.spark.scheduler.Task.run(Task.scala:88)<br>=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.spark.executor.Executor$TaskRunner.r=
un(Executor.scala:214)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at jav=
a.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145=
)<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.Thr=
eadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br>=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Thread.run(Thread.java:745)<br>Cau=
sed by: java.net.ConnectException: Connection refused: ip-xxx-xx-xx-xxx.ec2=
.internal/xxx.xx.xx.xxx:36684<br><br></div><div>Thanks,<br></div><div>Bhara=
th<br></div><div><br></div></div></div></div></div>

--001a11406d388ad6f705220363ad--