From user-return-254-archive-asf-public=cust-asf.ponee.io@livy.incubator.apache.org Tue Mar 20 11:44:33 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 09CA318064A for ; Tue, 20 Mar 2018 11:44:31 +0100 (CET) Received: (qmail 80994 invoked by uid 500); 20 Mar 2018 10:44:31 -0000 Mailing-List: contact user-help@livy.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@livy.incubator.apache.org Delivered-To: mailing list user@livy.incubator.apache.org Received: (qmail 80984 invoked by uid 99); 20 Mar 2018 10:44:30 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Mar 2018 10:44:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 7420AC2BE2 for ; Tue, 20 Mar 2018 10:44:30 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.63 X-Spam-Level: ** X-Spam-Status: No, score=2.63 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id aIHyKtC0bd_J for ; Tue, 20 Mar 2018 10:44:27 +0000 (UTC) Received: from mail-qk0-f173.google.com (mail-qk0-f173.google.com [209.85.220.173]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id EF1775F4EB for ; Tue, 20 Mar 2018 10:44:26 +0000 (UTC) Received: by mail-qk0-f173.google.com with SMTP id h14so1038033qkj.11 for ; Tue, 20 Mar 2018 03:44:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=q6f1fx9znX5onGCrY3j1bfU/qJXvEOMdGegcGaeelRA=; b=bVu5+HHfhASMtcwEhhuhS3iOK5VQLgnjbCWjQZHkC4LhH3JdjmrjjUBFZ3GJn8qgbJ G3zE8wTCFpiqMIFuoVHO6MzmfJlKilsbAYNzxaYqVlZVuWKV8LhGz/+7L2/QB25QE8mP wCB6lvKiVWSg9Z2VvDV9RfFFVyxAIx/rZN31trEN38d4vlseeC90y0EP8nnADRLx/JgJ 5/iJMOMFdSel8KRmMIZkDyJdlDbL8SqTbRRYZcBjsjpyBG75wHt6n0muT+IE45ETCNS/ NCFbb9Jnk8tMV2PW8GjEpJemcjHKNKDjMsTJBcrKND9CF9h37KfOBB/dhKJLg60MArf9 8clw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=q6f1fx9znX5onGCrY3j1bfU/qJXvEOMdGegcGaeelRA=; b=QwFMjhLDgpAGNDFkUJ7h4kRbk7gwVfiNwWEtdSBW4XTJnm3J78CL+1RQORKyVNhfJ8 DtkxcqcnblXWtUVwEdBknjfOrkS9XWP+D7ky2UO1wgKGCHf5Ud+436Cco5kUMj/92X2v o8X5CG/A2M3A8KiRd3E40FS2vj7xyIXa8HouQE/7gg8vT3yj35oFljnMNKMa75EtYZZ0 eglHKLcbPQQ2PXqFn4X3B43E43hT2dbQwN/QdDB+JixGyoMGFd2ZlHf+Ij5tqdI9BQaT 1TyOxTStIFX2Cqxe1B77A0GYYWZfzh6hJCY9iLbnt2CojHxyfeWdTf8ygwYi3YL5WQxQ /tYA== X-Gm-Message-State: AElRT7GyutVw7hawFVYkX7mCcuEcArINX03uGdKhzRw9iLSX6FIkUHy3 hkK/bpJ0oo/ZzFGJ7R/WxaYPd+TgWXKHXRabjPFlRw== X-Google-Smtp-Source: AG47ELuL1jBatel4TM2x2TxdtrsurPvmPd/TuSp8qdokni3teLE+zTKRXQXlWEpYtbxl94GCL1nJqkRkxn9sdzA6yRc= X-Received: by 10.55.43.144 with SMTP id r16mr22674558qkr.117.1521542665922; Tue, 20 Mar 2018 03:44:25 -0700 (PDT) MIME-Version: 1.0 Received: by 10.12.152.21 with HTTP; Tue, 20 Mar 2018 03:43:45 -0700 (PDT) From: kant kodali Date: Tue, 20 Mar 2018 03:43:45 -0700 Message-ID: Subject: Livy unable to recover spark session using yarn. using livy 0.4.0 & spark 2.3.0 & hadoop 2.7.2 To: user@livy.incubator.apache.org Content-Type: multipart/alternative; boundary="001a114716481cece90567d5c233" --001a114716481cece90567d5c233 Content-Type: text/plain; charset="UTF-8" Hi All, I am using Yarn and had set the following setting in livy.conf but Livy was unable to recover the session moreover the spark job spawned by Livy got killed when it is supposed to be running. Below is the stack trace. I am not sure what to deduce from it? The following are the settings I set in livy.conf that are needed for recovery. I did not set anything for "livy.spark.deploy-mode" (It's actually commented out) livy.spark.master = yarn livy.server.recovery.mode = recovery livy.server.recovery.state-store = zookeeper livy.server.recovery.state-store.url = localhost:2181 Here is my stacktrace 18/03/20 09:15:57 INFO ZooKeeper: Client environment:user.dir=/home/ubuntu/livy-0.4.0-incubating-bin/bin 18/03/20 09:15:57 INFO ZooKeeper: Initiating client connection, connectString=x.x.x.x:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@4ae9cfc1 18/03/20 09:15:57 INFO StateStore$: Using ZooKeeperStateStore for recovery. 18/03/20 09:15:57 INFO ClientCnxn: Opening socket connection to server x.x.x.x/x.x.x.x:2181. Will not attempt to authenticate using SASL (unknown error) 18/03/20 09:15:57 INFO ClientCnxn: Socket connection established to x.x.x.x/x.x.x.x:2181, initiating session 18/03/20 09:15:57 INFO ClientCnxn: Session establishment complete on server x.x.x.x/x.x.x.x:2181, sessionid = 0x162426cb5710007, negotiated timeout = 60000 18/03/20 09:15:57 INFO ConnectionStateManager: State change: CONNECTED 18/03/20 09:15:57 INFO BatchSessionManager: Recovered 0 batch sessions. Next session id: 0 18/03/20 09:15:58 ERROR RSCClient: Failed to connect to context. java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:748) 18/03/20 09:15:58 INFO InteractiveSessionManager: Recovered 1 interactive sessions. Next session id: 4 18/03/20 09:15:58 INFO InteractiveSessionManager: Registering new session 3 18/03/20 09:15:58 INFO InteractiveSessionManager: Heartbeat watchdog thread started. 18/03/20 09:15:58 ERROR RSCClient: RPC error. java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:748) 18/03/20 09:15:58 INFO WebServer: Starting server on http://x.x.x.x.us-west-2.compute.internal:8998 18/03/20 09:18:32 ERROR SessionServlet$: internal error java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) at org.apache.livy.server.interactive.InteractiveSession.statements(InteractiveSession.scala:471) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:103) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:102) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withViewAccessSession$1.apply(SessionHeartbeat.scala:69) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withViewAccessSession$1.apply(SessionHeartbeat.scala:67) at org.apache.livy.server.SessionServlet.doWithSession(SessionServlet.scala:221) at org.apache.livy.server.SessionServlet.withViewAccessSession(SessionServlet.scala:205) at org.apache.livy.server.interactive.InteractiveSessionServlet.org$apache$livy$server$interactive$SessionHeartbeatNotifier$$super$withViewAccessSession(InteractiveSessionServlet.scala:40) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$class.withViewAccessSession(SessionHeartbeat.scala:67) at org.apache.livy.server.interactive.InteractiveSessionServlet.withViewAccessSession(InteractiveSessionServlet.scala:40) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7.apply(InteractiveSessionServlet.scala:102) at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270) at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:178) at org.apache.livy.server.JsonServlet.withRouteMultiParams(JsonServlet.scala:39) at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264) at org.scalatra.ScalatraServlet.invoke(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:240) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:238) at scala.Option.flatMap(Option.scala:170) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:238) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:237) at scala.collection.immutable.Stream.flatMap(Stream.scala:446) at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237) at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163) at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(ScalatraBase.scala:175) at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193) at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175) at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBase.scala:113) at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111) at org.scalatra.ScalatraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraServlet.scala:49) at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43) at org.apache.livy.server.SessionServlet.org$scalatra$MethodOverride$$super$handle(SessionServlet.scala:39) at org.scalatra.MethodOverride$class.handle(MethodOverride.scala:28) at org.apache.livy.server.SessionServlet.org$scalatra$GZipSupport$$super$handle(SessionServlet.scala:39) at org.scalatra.GZipSupport$$anonfun$handle$1.apply$mcV$sp(GZipSupport.scala:34) at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19) at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) at org.scalatra.GZipSupport$class.handle(GZipSupport.scala:18) at org.apache.livy.server.interactive.InteractiveSessionServlet.org$scalatra$servlet$FileUploadSupport$$super$handle(InteractiveSessionServlet.scala:40) at org.scalatra.servlet.FileUploadSupport$class.handle(FileUploadSupport.scala:93) at org.apache.livy.server.interactive.InteractiveSessionServlet.handle(InteractiveSessionServlet.scala:40) at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) ... 1 more 18/03/20 09:18:36 ERROR SessionServlet$: internal error java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) at org.apache.livy.server.interactive.InteractiveSession.statements(InteractiveSession.scala:471) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:103) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:102) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withViewAccessSession$1.apply(SessionHeartbeat.scala:69) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withViewAccessSession$1.apply(SessionHeartbeat.scala:67) at org.apache.livy.server.SessionServlet.doWithSession(SessionServlet.scala:221) at org.apache.livy.server.SessionServlet.withViewAccessSession(SessionServlet.scala:205) at org.apache.livy.server.interactive.InteractiveSessionServlet.org$apache$livy$server$interactive$SessionHeartbeatNotifier$$super$withViewAccessSession(InteractiveSessionServlet.scala:40) at org.apache.livy.server.interactive.SessionHeartbeatNotifier$class.withViewAccessSession(SessionHeartbeat.scala:67) at org.apache.livy.server.interactive.InteractiveSessionServlet.withViewAccessSession(InteractiveSessionServlet.scala:40) at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$7.apply(InteractiveSessionServlet.scala:102) at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270) at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:178) at org.apache.livy.server.JsonServlet.withRouteMultiParams(JsonServlet.scala:39) at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264) at org.scalatra.ScalatraServlet.invoke(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:240) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:238) at scala.Option.flatMap(Option.scala:170) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:238) at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:237) at scala.collection.immutable.Stream.flatMap(Stream.scala:446) at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237) at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163) at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(ScalatraBase.scala:175) at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193) at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175) at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBase.scala:113) at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111) at org.scalatra.ScalatraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraServlet.scala:49) at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43) at org.apache.livy.server.SessionServlet.org$scalatra$MethodOverride$$super$handle(SessionServlet.scala:39) at org.scalatra.MethodOverride$class.handle(MethodOverride.scala:28) at org.apache.livy.server.SessionServlet.org$scalatra$GZipSupport$$super$handle(SessionServlet.scala:39) at org.scalatra.GZipSupport$$anonfun$handle$1.apply$mcV$sp(GZipSupport.scala:34) at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19) at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) at org.scalatra.GZipSupport$class.handle(GZipSupport.scala:18) at org.apache.livy.server.interactive.InteractiveSessionServlet.org$scalatra$servlet$FileUploadSupport$$super$handle(InteractiveSessionServlet.scala:40) at org.scalatra.servlet.FileUploadSupport$class.handle(FileUploadSupport.scala:93) at org.apache.livy.server.interactive.InteractiveSessionServlet.handle(InteractiveSessionServlet.scala:40) at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) ... 1 more --001a114716481cece90567d5c233 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi All,

I am using Yarn and had set the= following setting in livy.conf but Livy was unable to recover the session = moreover the spark job spawned by Livy got killed when it is supposed to be= running.=C2=A0 Below is the stack trace. I am not sure what to deduce from= it?

The following are the settings I set in=C2=A0= livy.conf that are needed for recovery. I did not set anything for "livy.spark.deploy-mode" (It's actu= ally commented out)

livy.spark.master =3D=C2=A0yarn

livy.server.recovery.mode =3D recovery<= /span>

livy.server.recovery.state-store =3D zookeeper<= br>

livy.server.recovery.state-store.ur= l =3D localhost:2181



Here is my stacktrace

= 
18/03/20 09:15:57 INFO ZooKeeper: Client env=
ironment:user.dir=3D/home/ubuntu/livy-0.4.0-incubating-bin/bin
18/03/20 09:15:57 INFO ZooKeeper: Initiating client connection, connectStri=
ng=3Dx.x.x.x:2181 sessionTimeout=3D60000 watcher=3Dorg.apache.curator.Conne=
ctionState@4ae9cfc1
18/03/20 09:15:57 INFO StateStore$: Using ZooKeeperStateStore for recovery.
18/03/20 09:15:57 INFO ClientCnxn: Opening socket connection to server x.x.=
x.x/x.x.x.x:2181. Will not attempt to authenticate using SASL (unknown erro=
r)
18/03/20 09:15:57 INFO ClientCnxn: Socket connection established to x.x.x.x=
/x.x.x.x:2181, initiating session
18/03/20 09:15:57 INFO ClientCnxn: Session establishment complete on server=
 x.x.x.x/x.x.x.x:2181, sessionid =3D 0x162426cb5710007, negotiated timeout =
=3D 60000
18/03/20 09:15:57 INFO ConnectionStateManager: State change: CONNECTED
18/03/20 09:15:57 INFO BatchSessionManager: Recovered 0 batch sessions. Nex=
t session id: 0
18/03/20 09:15:58 ERROR RSCClient: Failed to connect to context.
java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.in=
ternal/x.x.x.x:10001
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:71=
7)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSock=
etChannel.java:224)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConn=
ect(AbstractNioChannel.java:289)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.ja=
va:528)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEv=
entLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.j=
ava:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThrea=
dEventExecutor.java:111)
    at java.lang.Thread.run(Thread.java:748)
18/03/20 09:15:58 INFO InteractiveSessionManager: Recovered 1 interactive s=
essions. Next session id: 4
18/03/20 09:15:58 INFO InteractiveSessionManager: Registering new session 3
18/03/20 09:15:58 INFO InteractiveSessionManager: Heartbeat watchdog thread=
 started.
18/03/20 09:15:58 ERROR RSCClient: RPC error.
java.net.ConnectException: Connection refused: x.x.x.x.us-west-2.compute.in=
ternal/x.x.x.x:10001
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:71=
7)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSock=
etChannel.java:224)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConn=
ect(AbstractNioChannel.java:289)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.ja=
va:528)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEv=
entLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.j=
ava:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThrea=
dEventExecutor.java:111)
    at java.lang.Thread.run(Thread.java:748)
18/03/20 09:15:58 INFO WebServer: Starting server on http://x.x.x.x.us-west-2.compute.inter=
nal:8998
18/03/20 09:18:32 ERROR SessionServlet$: internal error
java.util.concurrent.ExecutionException: java.net.ConnectException: Connect=
ion refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001
    at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
    at org.apache.livy.server.interactive.InteractiveSession.statements(Int=
eractiveSession.scala:471)
    at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfu=
n$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:103)
    at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfu=
n$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:102)
    at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun=
$withViewAccessSession$1.apply(SessionHeartbeat.scala:69)
    at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun=
$withViewAccessSession$1.apply(SessionHeartbeat.scala:67)
    at org.apache.livy.server.SessionServlet.doWithSession(SessionServlet.s=
cala:221)
    at org.apache.livy.server.SessionServlet.withViewAccessSession(SessionS=
ervlet.scala:205)
    at org.apache.livy.server.interactive.InteractiveSessionServle=
t.org$apache$livy$server$interactive$SessionHeartbeatNotifier$$super$wi=
thViewAccessSession(InteractiveSessionServlet.scala:40)
    at org.apache.livy.server.interactive.SessionHeartbeatNotifier$class.wi=
thViewAccessSession(SessionHeartbeat.scala:67)
    at org.apache.livy.server.interactive.InteractiveSessionServlet.withVie=
wAccessSession(InteractiveSessionServlet.scala:40)
    at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfu=
n$7.apply(InteractiveSessionServlet.scala:102)
    at org.scalatra.ScalatraBase$class.org=
$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270)
    at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala=
:265)
    at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala=
:265)
    at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:=
178)
    at org.apache.livy.server.JsonServlet.withRouteMultiParams(JsonServlet.=
scala:39)
    at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264)
    at org.scalatra.ScalatraServlet.invoke(ScalatraServlet.scala:49)
    at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.appl=
y(ScalatraBase.scala:240)
    at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.appl=
y(ScalatraBase.scala:238)
    at scala.Option.flatMap(Option.scala:170)
    at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.sc=
ala:238)
    at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.sc=
ala:237)
    at scala.collection.immutable.Stream.flatMap(Stream.scala:446)
    at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237)
    at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49)
    at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163)
    at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(Scal=
atraBase.scala:175)
    at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBas=
e.scala:175)
    at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBas=
e.scala:175)
    at org.scalatra.ScalatraBase$class.org=
$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193)
    at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175=
)
    at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49)
    at org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBas=
e.scala:113)
    at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala=
:113)
    at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala=
:113)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
    at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80)
    at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49)
    at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(Dynam=
icScope.scala:60)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
    at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71)
    at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49)
    at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.sca=
la:59)
    at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.sca=
la:49)
    at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111)
    at org.scalatra.Sca=
latraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraSer=
vlet.scala:49)
    at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43)
    at org.apa=
che.livy.server.SessionServlet.org$scalatra$MethodOverride$$super$handl=
e(SessionServlet.scala:39)
    at org.scalatra.MethodOverride$class.handle(MethodOverride.scala:28)
    at org.apa=
che.livy.server.SessionServlet.org$scalatra$GZipSupport$$super$handle(S=
essionServlet.scala:39)
    at org.scalatra.GZipSupport$$anonfun$handle$1.apply$mcV$sp(GZipSupport.=
scala:34)
    at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:1=
9)
    at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:1=
9)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
    at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80)
    at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49)
    at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(Dynam=
icScope.scala:60)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
    at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71)
    at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49)
    at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.sca=
la:59)
    at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.sca=
la:49)
    at org.scalatra.GZipSupport$class.handle(GZipSupport.scala:18)
    at org.apache.livy.server.interactive.InteractiveSessionServle=
t.org$scalatra$servlet$FileUploadSupport$$super$handle(InteractiveSessi=
onServlet.scala:40)
    at org.scalatra.servlet.FileUploadSupport$class.handle(FileUploadSuppor=
t.scala:93)
    at org.apache.livy.server.interactive.InteractiveSessionServlet.handle(=
InteractiveSessionServlet.scala:40)
    at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:81=
2)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.jav=
a:587)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHand=
ler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java=
:515)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandl=
er.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.=
java:141)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCol=
lection.java:110)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrappe=
r.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:499)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.ja=
va:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.jav=
a:544)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPo=
ol.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPoo=
l.java:555)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused: x.x.x.x.us-west-2=
.compute.internal/x.x.x.x:10001
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:71=
7)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSock=
etChannel.java:224)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConn=
ect(AbstractNioChannel.java:289)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.ja=
va:528)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEv=
entLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.j=
ava:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThrea=
dEventExecutor.java:111)
    ... 1 more
18/03/20 09:18:36 ERROR SessionServlet$: internal error
java.util.concurrent.ExecutionException: java.net.ConnectException: Connect=
ion refused: x.x.x.x.us-west-2.compute.internal/x.x.x.x:10001
    at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
    at org.apache.livy.server.interactive.InteractiveSession.statements(Int=
eractiveSession.scala:471)
    at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfu=
n$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:103)
    at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfu=
n$7$$anonfun$apply$3.apply(InteractiveSessionServlet.scala:102)
    at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun=
$withViewAccessSession$1.apply(SessionHeartbeat.scala:69)
    at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun=
$withViewAccessSession$1.apply(SessionHeartbeat.scala:67)
    at org.apache.livy.server.SessionServlet.doWithSession(SessionServlet.s=
cala:221)
    at org.apache.livy.server.SessionServlet.withViewAccessSession(SessionS=
ervlet.scala:205)
    at org.apache.livy.server.interactive.InteractiveSessionServle=
t.org$apache$livy$server$interactive$SessionHeartbeatNotifier$$super$wi=
thViewAccessSession(InteractiveSessionServlet.scala:40)
    at org.apache.livy.server.interactive.SessionHeartbeatNotifier$class.wi=
thViewAccessSession(SessionHeartbeat.scala:67)
    at org.apache.livy.server.interactive.InteractiveSessionServlet.withVie=
wAccessSession(InteractiveSessionServlet.scala:40)
    at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfu=
n$7.apply(InteractiveSessionServlet.scala:102)
    at org.scalatra.ScalatraBase$class.org=
$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270)
    at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala=
:265)
    at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala=
:265)
    at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:=
178)
    at org.apache.livy.server.JsonServlet.withRouteMultiParams(JsonServlet.=
scala:39)
    at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264)
    at org.scalatra.ScalatraServlet.invoke(ScalatraServlet.scala:49)
    at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.appl=
y(ScalatraBase.scala:240)
    at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.appl=
y(ScalatraBase.scala:238)
    at scala.Option.flatMap(Option.scala:170)
    at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.sc=
ala:238)
    at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.sc=
ala:237)
    at scala.collection.immutable.Stream.flatMap(Stream.scala:446)
    at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237)
    at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49)
    at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163)
    at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(Scal=
atraBase.scala:175)
    at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBas=
e.scala:175)
    at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBas=
e.scala:175)
    at org.scalatra.ScalatraBase$class.org=
$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193)
    at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175=
)
    at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49)
    at org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBas=
e.scala:113)
    at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala=
:113)
    at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala=
:113)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
    at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80)
    at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49)
    at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(Dynam=
icScope.scala:60)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
    at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71)
    at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49)
    at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.sca=
la:59)
    at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.sca=
la:49)
    at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111)
    at org.scalatra.Sca=
latraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraSer=
vlet.scala:49)
    at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43)
    at org.apa=
che.livy.server.SessionServlet.org$scalatra$MethodOverride$$super$handl=
e(SessionServlet.scala:39)
    at org.scalatra.MethodOverride$class.handle(MethodOverride.scala:28)
    at org.apa=
che.livy.server.SessionServlet.org$scalatra$GZipSupport$$super$handle(S=
essionServlet.scala:39)
    at org.scalatra.GZipSupport$$anonfun$handle$1.apply$mcV$sp(GZipSupport.=
scala:34)
    at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:1=
9)
    at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:1=
9)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
    at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80)
    at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49)
    at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(Dynam=
icScope.scala:60)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
    at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71)
    at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49)
    at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.sca=
la:59)
    at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.sca=
la:49)
    at org.scalatra.GZipSupport$class.handle(GZipSupport.scala:18)
    at org.apache.livy.server.interactive.InteractiveSessionServle=
t.org$scalatra$servlet$FileUploadSupport$$super$handle(InteractiveSessi=
onServlet.scala:40)
    at org.scalatra.servlet.FileUploadSupport$class.handle(FileUploadSuppor=
t.scala:93)
    at org.apache.livy.server.interactive.InteractiveSessionServlet.handle(=
InteractiveSessionServlet.scala:40)
    at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:81=
2)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.jav=
a:587)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHand=
ler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java=
:515)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandl=
er.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.=
java:141)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCol=
lection.java:110)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrappe=
r.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:499)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.ja=
va:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.jav=
a:544)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPo=
ol.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPoo=
l.java:555)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused: x.x.x.x.us-west-2=
.compute.internal/x.x.x.x:10001
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:71=
7)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSock=
etChannel.java:224)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConn=
ect(AbstractNioChannel.java:289)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.ja=
va:528)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEv=
entLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.j=
ava:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThrea=
dEventExecutor.java:111)
    ... 1 more
--001a114716481cece90567d5c233--