livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Junaid Nasir <jna...@an10.io>
Subject Re: Livy with DC/OS Mesos
Date Wed, 25 Oct 2017 13:06:55 GMT
I am able to run spark shell from command line using command
/spark/dist/bin/pyspark --master mesos://leader.mesos:5050 --conf
spark.mesos.executor.docker.image=mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6
--conf spark.executor.home=/opt/spark/dist
I have changed livy's configuration accordingly i.e set master to leader.mesos
when i send post request to /sessions with input data.{'kind': 'spark', 'conf':
{'spark.mesos.executor.docker.image':
'mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6', 'spark.executor.home':
'/opt/spark/dist'}}it produces these errors on livy console

                  17/10/25 13:00:00 INFO InteractiveSession$: Creating Interactive session
0: [owner: null, request: [kind: spark, proxyUser: None, conf: spark.mesos.executor.docker.image
-> mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6,spark.executor.home -> /opt/spark/dist,
heartbeatTimeoutInSecond: 0]]17/10/25 13:00:00 INFO RpcServer: Connected to the port 1000017/10/25
13:00:00 WARN RSCConf: Your hostname, agent0005.c.calm-airship-162807.internal, resolves to
a loopback address, but we couldn't find any external IP address!17/10/25 13:00:00 WARN RSCConf:
Set livy.rsc.rpc.server.address if you need to bind to another address.17/10/25 13:00:00 INFO
LineBufferedStream: stdout: spark-env: User: root17/10/25 13:00:00 INFO LineBufferedStream:
stdout: Skipping bootstrap IP detection17/10/25 13:00:00 INFO LineBufferedStream: stdout:
spark-env: No kerberos KDC config found17/10/25 13:00:00 INFO InteractiveSessionManager: Registering
new session 017/10/25 13:00:01 INFO LineBufferedStream: stdout: 17/10/25 13:00:01 WARN NativeCodeLoader:
Unable to load native-hadoop library for your platform... using builtin-java classes where
applicable17/10/25 13:00:01 INFO LineBufferedStream: stdout: 17/10/25 13:00:01 INFO RSCDriver:
Connecting to: agent0005.c.calm-airship-162807.internal:1000017/10/25 13:00:01 INFO LineBufferedStream:
stdout: 17/10/25 13:00:01 INFO RSCDriver: Starting RPC server...17/10/25 13:00:01 INFO LineBufferedStream:
stdout: 17/10/25 13:00:01 INFO RpcServer: Connected to the port 1000117/10/25 13:00:01 INFO
LineBufferedStream: stdout: 17/10/25 13:00:01 WARN RSCConf: Your hostname, agent0005.c.calm-airship-162807.internal,
resolves to a loopback address, but we couldn't find any external IP address!17/10/25 13:00:01
INFO LineBufferedStream: stdout: 17/10/25 13:00:01 WARN RSCConf: Set livy.rsc.rpc.server.address
if you need to bind to another address.17/10/25 13:00:02 INFO LineBufferedStream: stdout:
17/10/25 13:00:02 INFO RSCDriver: Received job request fcbdd93f-fcd7-453d-9d0f-c303a687c7ae17/10/25
13:00:02 INFO LineBufferedStream: stdout: 17/10/25 13:00:02 INFO RSCDriver: SparkContext not
yet up, queueing job request.17/10/25 13:00:04 WARN RSCClient: Client RPC channel closed unexpectedly.17/10/25
13:00:04 WARN ContextLauncher: Child process exited with code 137.17/10/25 13:00:04 WARN RSCClient:
Error stopping RPC.io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@1831bad3(uncancellable)
       at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390) 
      at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
       at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251)        at
io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)        at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)
       at io.netty.util.concurrent.DefaultPromise.sync(DefaultPromise.java:218)        at
io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:117)        at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:28)
       at org.apache.livy.rsc.rpc.Rpc.close(Rpc.java:307)        at org.apache.livy.rsc.RSCClient.stop(RSCClient.java:232)
       at org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:129)        at org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:123)
       at org.apache.livy.rsc.Utils$2.operationComplete(Utils.java:108)        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
       at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
       at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)    
   at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)   
    at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:956)  
     at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:608)
       at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:586)
       at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
       at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158)
       at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)    
   at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
       at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)   
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
       at java.lang.Thread.run(Thread.java:745)17/10/25 13:00:04 ERROR SparkProcApp: spark-submit
exited with code 13717/10/25 13:00:04 INFO RSCClient: Failing pending job fcbdd93f-fcd7-453d-9d0f-c303a687c7ae
due to shutdown.17/10/25 13:00:04 INFO InteractiveSession: Failed to ping RSC driver for session
0. Killing application.17/10/25 13:00:04 INFO InteractiveSession: Stopping InteractiveSession
0...17/10/25 13:00:04 INFO InteractiveSession: Stopped InteractiveSession 0.
                

Not using Mixmax yet?  
i noticed the line Set livy.rsc.rpc.server.address if you need to bind to
another address. have tried setting this in conf/livy-client with livy's server
local ip address. but error remains.
any help would be great
Thanks for your time  





On Wed, Oct 25, 2017 5:47 PM, Junaid Nasir jnasir@an10.io  wrote:
livy version 0.4trying to create interactive session. may be it's not supported
with mesos?  





On Wed, Oct 25, 2017 3:55 PM, Junaid Nasir jnasir@an10.io  wrote:
Hi,
I am trying to run Livy with DC/OS Mesos.it tries to create driver but that
fails with error blow. Is there a way i can provide it with spark-internal file?
I am unable to find what this spark-internal file does and i am unable to find
it anywhere. only thing i found related to this was in ContextLauncher.java line
232 ` launcher.setAppResource("spark-internal");`

Failed to fetch spark-internal                   I1025 10:41:45.202145 27901 fetcher.cpp:531]
Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/babc8653-5a13-4fca-9fd9-fe2870f9ad50-S14","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"spark-internal"}}],"sandbox_directory":"\/var\/lib\/mesos\/slave\/slaves\/babc8653-5a13-4fca-9fd9-fe2870f9ad50-S14\/frameworks\/babc8653-5a13-4fca-9fd9-fe2870f9ad50-0003\/executors\/driver-20171025104145-0034\/runs\/9976b5de-0ed7-428e-a23c-59056c6f29ed"}I1025
10:41:45.204314 27901 fetcher.cpp:442] Fetching URI 'spark-internal'I1025 10:41:45.204329
27901 fetcher.cpp:283] Fetching directly into the sandbox directoryI1025 10:41:45.204346 27901
fetcher.cpp:220] Fetching URI 'spark-internal'Failed to fetch 'spark-internal': A relative
path was passed for the resource but the Mesos framework home was not specified. Please either
provide this config option or avoid using a relative path
                

Not using Mixmax yet?  

Spark-submit works fine I am unable to run a job with following command
bin/spark-submit --class org.apache.spark.examples.SparkPi --master
mesos://10.128.0.23:22401 --deploy-mode cluster --supervise --executor-memory 1G
--total-executor-cores 1 --conf
spark.mesos.executor.docker.image=mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6
--conf spark.executor.home=/opt/spark/dist
https://downloads.mesosphere.com/spark/examples/pi.py 30
I have set spark master to the same ip and set cluster mode. sending post
request to session with these parametersdata = {'kind':
'spark','conf':{'spark.mesos.executor.docker.image':'mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6
','spark.executor.home':'/opt/spark/dist'}} 
Any help would be highlyappreciated
Mime
View raw message