Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D9BEF1813B for ; Thu, 15 Oct 2015 00:21:03 +0000 (UTC) Received: (qmail 77560 invoked by uid 500); 15 Oct 2015 00:20:59 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 77445 invoked by uid 500); 15 Oct 2015 00:20:59 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 77433 invoked by uid 99); 15 Oct 2015 00:20:59 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Oct 2015 00:20:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 8D4BFC0FC1 for ; Thu, 15 Oct 2015 00:20:58 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.901 X-Spam-Level: ** X-Spam-Status: No, score=2.901 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id G3OYL7mw-gtJ for ; Thu, 15 Oct 2015 00:20:47 +0000 (UTC) Received: from mail-qg0-f48.google.com (mail-qg0-f48.google.com [209.85.192.48]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id D5C5324E1C for ; Thu, 15 Oct 2015 00:20:46 +0000 (UTC) Received: by qgeo38 with SMTP id o38so14183986qge.0 for ; Wed, 14 Oct 2015 17:20:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=1pqcCyU/DHwukcAQmcFSJ648SbIv1HDf87ikbIp0bPc=; b=Ml5bAt4zCkTYBH2V5eG2Toc6OngOlTkNYrq1HumgR7QIscxpnuiYv134KUgBbcD+YA WdsX8af8/zBhSxhueAGwMtrpXGvgnpSDyfM8mNgGOLzL7u8PEg/zahanwpNedg/9Jxiw Zvi759JGsk6XrFi/aV8EJO32K7mxAriFT/peCwl+fyCDd/linSKcssxJpHnjegKgWWJZ yKOExsspxEGvkShz1xUDWZ1vhQ0ObHBPn34U59How+2lmtKqpOQ2ewp0BKnuULc2dEA+ hTWhRJShqT1oYMNpo34FveScewKB1oAXH7i7JqPQJJKJuypmZ7qHW34XoDWz1CYtAETC wQYg== MIME-Version: 1.0 X-Received: by 10.140.39.168 with SMTP id v37mr7692387qgv.24.1444868445804; Wed, 14 Oct 2015 17:20:45 -0700 (PDT) Received: by 10.140.96.182 with HTTP; Wed, 14 Oct 2015 17:20:45 -0700 (PDT) Received: by 10.140.96.182 with HTTP; Wed, 14 Oct 2015 17:20:45 -0700 (PDT) In-Reply-To: References: Date: Thu, 15 Oct 2015 05:50:45 +0530 Message-ID: Subject: Re: Spark Master Dying saying TimeoutException From: Raghavendra Pandey To: Kartik Mathur Cc: User Content-Type: multipart/alternative; boundary=001a11c1286e75b2cc052219a6ad --001a11c1286e75b2cc052219a6ad Content-Type: text/plain; charset=UTF-8 I fixed these timeout errors by retrying... On Oct 15, 2015 3:41 AM, "Kartik Mathur" wrote: > Hi, > > I have some nightly jobs which runs every night but dies sometimes because > of unresponsive master , spark master logs says - > > Not seeing much else there , what could possible cause an exception like > this. > > *Exception in thread "main" java.util.concurrent.TimeoutException: Futures > timed out after [10000 milliseconds]* > > at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > at scala.concurrent.Await$.result(package.scala:107) > > at akka.remote.Remoting.start(Remoting.scala:180) > > at > akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184) > > at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:618) > > at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:615) > > at akka.actor.ActorSystemImpl._start(ActorSystem.scala:615) > > at akka.actor.ActorSystemImpl.start(ActorSystem.scala:632) > > at akka.actor.ActorSystem$.apply(ActorSystem.scala:141) > > 2015-10-14 05:43:04 ERROR Remoting:65 - Remoting error: [Startup timed > out] [ > > akka.remote.RemoteTransportException: Startup timed out > > at > akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:136) > > at akka.remote.Remoting.start(Remoting.scala:198) > > at > akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184) > > at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:618) > > at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:615) > > at akka.actor.ActorSystemImpl._start(ActorSystem.scala:615) > > at akka.actor.ActorSystemImpl.start(ActorSystem.scala:632) > > at akka.actor.ActorSystem$.apply(ActorSystem.scala:141) > > at akka.actor.ActorSystem$.apply(ActorSystem.scala:118) > > at > org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:122) > > at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:55) > > at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54) > > at > org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1837) > > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) > > at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1828) > > at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:57) > > at > org.apache.spark.deploy.master.Master$.startSystemAndActor(Master.scala:906) > > at org.apache.spark.deploy.master.Master$.main(Master.scala:869) > > at org.apache.spark.deploy.master.Master.main(Master.scala) > > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [10000 milliseconds] > > at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > > at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > > at scala.concurrent.Await$.result(package.scala:107) > > at akka.remote.Remoting.start(Remoting.scala:180) > > ... 17 more > > > --001a11c1286e75b2cc052219a6ad Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

I fixed these timeout errors by retrying...

On Oct 15, 2015 3:41 AM, "Kartik Mathur&quo= t; <kartik@bluedata.com> w= rote:
Hi,

I have some nightly jobs which runs every night b= ut dies sometimes because of unresponsive master , spark master logs says -= =C2=A0

Not seeing much else there , what could pos= sible cause an exception like this.

Exception in thread "main" java.util.concurrent.Timeo= utException: Futures timed out after [10000 milliseconds]

at scala.concurrent.impl.Promise$De= faultPromise.ready(Promise.scala:219)

at scala.concurrent.impl.Promise$De= faultPromise.result(Promise.scala:223)

at scala.concurrent.Await$$anonfun$= result$1.apply(package.scala:107)

at scala.concurrent.BlockContext$De= faultBlockContext$.blockOn(BlockContext.scala:53)

at scala.concurrent.Await$.result(p= ackage.scala:107)

at akka.remote.Remoting.start(Remot= ing.scala:180)

at akka.remote.RemoteActorRefProvid= er.init(RemoteActorRefProvider.scala:184)

at akka.actor.ActorSystemImpl.lifte= dTree2$1(ActorSystem.scala:618)

at akka.actor.ActorSystemImpl._star= t$lzycompute(ActorSystem.scala:615)

at akka.actor.ActorSystemImpl._star= t(ActorSystem.scala:615)

at akka.actor.ActorSystemImpl.start= (ActorSystem.scala:632)

at akka.actor.ActorSystem$.apply(Ac= torSystem.scala:141)

2015-10-14 05:43:04 ERROR Remoting:65 - Remoting = error: [Startup timed out] [

akka.remote.RemoteTransportException: Startup tim= ed out

at akka.remote.Remoting.akka$remote= $Remoting$$notifyError(Remoting.scala:136)

at akka.remote.Remoting.start(Remot= ing.scala:198)

at akka.remote.RemoteActorRefProvid= er.init(RemoteActorRefProvider.scala:184)

at akka.actor.ActorSystemImpl.lifte= dTree2$1(ActorSystem.scala:618)

at akka.actor.ActorSystemImpl._star= t$lzycompute(ActorSystem.scala:615)

at akka.actor.ActorSystemImpl._star= t(ActorSystem.scala:615)

at akka.actor.ActorSystemImpl.start= (ActorSystem.scala:632)

at akka.actor.ActorSystem$.apply(Ac= torSystem.scala:141)

at akka.actor.ActorSystem$.apply(Ac= torSystem.scala:118)

at org.apache.spark.util.AkkaUtils$= .org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:122)<= /font>

at org.apache.spark.util.AkkaUtils$= $anonfun$1.apply(AkkaUtils.scala:55)

at org.apache.spark.util.AkkaUtils$= $anonfun$1.apply(AkkaUtils.scala:54)

at org.apache.spark.util.Utils$$ano= nfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1837)

at scala.collection.immutable.Range= .foreach$mVc$sp(Range.scala:141)

at org.apache.spark.util.Utils$.sta= rtServiceOnPort(Utils.scala:1828)

at org.apache.spark.util.AkkaUtils$= .createActorSystem(AkkaUtils.scala:57)

at org.apache.spark.deploy.master.M= aster$.startSystemAndActor(Master.scala:906)

at org.apache.spark.deploy.master.M= aster$.main(Master.scala:869)

at org.apache.spark.deploy.master.M= aster.main(Master.scala)

Caused by: java.util.concurrent.TimeoutException:= Futures timed out after [10000 milliseconds]

at scala.concurrent.impl.Promise$De= faultPromise.ready(Promise.scala:219)

at scala.concurrent.impl.Promise$De= faultPromise.result(Promise.scala:223)

at scala.concurrent.Await$$anonfun$= result$1.apply(package.scala:107)

at scala.concurrent.BlockContext$De= faultBlockContext$.blockOn(BlockContext.scala:53)

at scala.concurrent.Await$.result(p= ackage.scala:107)

at akka.remote.Remoting.start(Remot= ing.scala:180)

... 17 more

=


--001a11c1286e75b2cc052219a6ad--