livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Junaid Nasir <jna...@an10.io>
Subject Re: NoSuchFileException: spark-internal - when creating interactive session on remote spark
Date Mon, 16 Oct 2017 18:28:33 GMT
I followed https://livy.incubator.apache.org/examples/ to setup session for
pyspark using python. will any logs be useful to debug? also any plan to support
stand alone spark?I will try this with DC/OS spark deployment tomorrow. will
update this thread if that works or not  





On Mon, Oct 16, 2017 11:16 AM, Saisai Shao sai.sai.shao@gmail.com  wrote:
Would you please provide more information about how you create a Livy session?
As for now, Livy only supports spark on yarn and local mode officially, we don't
test on standalone cluster mode, so maybe there's some issues in it.
On Mon, Oct 16, 2017 at 4:29 AM, Junaid Nasir <jnasir@an10.io>  wrote:
Hi everyone,I am using livy (livy-0.4.0-incubating-bin), and a remote spark
(spark-2.2.0-bin-hadoop2.7) stand alone cluster. livy's setting is default
except setting livy.spark.master = spark://10.128.1.1:6066  and
livy.spark.deploy-mode = cluster (have tried with default deploy-mode but then
it throws a different error)when i try to create an interactive spark session on
remote cluster it creates the driver and immediately sends it to error state.  
livy console error                   17/10/15 20:18:08 INFO LineBufferedStream: stdout: 17/10/15
20:18:08 INFO RestSubmissionClient: Submitting a request to launch an application in spark://10.128.1.1:6066.17/10/15
20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO RestSubmissionClient: Submission
successfully created as driver-20171015201808-0002. Polling submission state...17/10/15 20:18:09
INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO RestSubmissionClient: Submitting a
request for the status of submission driver-20171015201808-0002 in spark://10.128.1.1:6066.17/10/15
20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO RestSubmissionClient: State
of driver driver-20171015201808-0002 is now ERROR.17/10/15 20:18:09 INFO LineBufferedStream:
stdout: 17/10/15 20:18:09 INFO RestSubmissionClient: Driver is running on worker worker-20171015195836-10.128.1.1-38097
at 10.128.1.1:38097.17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 ERROR
RestSubmissionClient: Exception from the cluster:17/10/15 20:18:09 INFO LineBufferedStream:
stdout: java.nio.file.NoSuchFileException: spark-internal17/10/15 20:18:09 INFO LineBufferedStream:
stdout:      sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)17/10/15
20:18:09 INFO LineBufferedStream: stdout:      sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)17/10/15
20:18:09 INFO LineBufferedStream: stdout:      sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)17/10/15
20:18:09 INFO LineBufferedStream: stdout:      sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)17/10/15
20:18:09 INFO LineBufferedStream: stdout:      sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)17/10/15
20:18:09 INFO LineBufferedStream: stdout:      java.nio.file.Files.copy(Files.java:1274)17/10/15
20:18:09 INFO LineBufferedStream: stdout:      org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:625)17/10/15
20:18:09 INFO LineBufferedStream: stdout:      org.apache.spark.util.Utils$.copyFile(Utils.scala:596)17/10/15
20:18:09 INFO LineBufferedStream: stdout:      org.apache.spark.util.Utils$.doFetchFile(Utils.scala:681)17/10/15
20:18:09 INFO LineBufferedStream: stdout:      org.apache.spark.util.Utils$.fetchFile(Utils.scala:480)17/10/15
20:18:09 INFO LineBufferedStream: stdout:      org.apache.spark.deploy.worker.DriverRunner.downloadUserJar(DriverRunner.scala:155)17/10/15
20:18:09 INFO LineBufferedStream: stdout:      org.apache.spark.deploy.worker.DriverRunner.prepareAndRunDriver(DriverRunner.scala:173)17/10/15
20:18:09 INFO LineBufferedStream: stdout:      org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:92)17/10/15
20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO RestSubmissionClient: Server
responded with CreateSubmissionResponse:17/10/15 20:18:09 INFO LineBufferedStream: stdout:
{17/10/15 20:18:09 INFO LineBufferedStream: stdout:   "action" : "CreateSubmissionResponse",17/10/15
20:18:09 INFO LineBufferedStream: stdout:   "message" : "Driver successfully submitted as
driver-20171015201808-0002",17/10/15 20:18:09 INFO LineBufferedStream: stdout:   "serverSparkVersion"
: "2.2.0",17/10/15 20:18:09 INFO LineBufferedStream: stdout:   "submissionId" : "driver-20171015201808-0002",17/10/15
20:18:09 INFO LineBufferedStream: stdout:   "success" : true17/10/15 20:18:09 INFO LineBufferedStream:
stdout: }
                

Not using Mixmax yet?  
can anyone please guide me how to resolve this?
I have tried setting up spark locally and it works fine. so i guess problem is
with uploading jar files to remote cluster. I am using GCE for both spark
cluster and livy server but they have all ports (on internal network open)
spark worker log show this too  
                  17/10/15 20:18:08 INFO Worker: Asked to launch driver driver-20171015201808-000217/10/15
20:18:08 INFO DriverRunner: Copying user jar spark-internal to /home/junaid.nasir/spark-2.2.0-bin-hadoop2.7/work/driver-20171015201808-0002/spark-internal17/10/15
20:18:08 INFO Utils: Copying /spark-internal to /home/junaid.nasir/spark-2.2.0-bin-hadoop2.7/work/driver-20171015201808-0002/spark-internal17/10/15
20:18:08 INFO DriverRunner: Killing driver process!17/10/15 20:18:08 WARN Worker: Driver driver-20171015201808-0002
failed with unrecoverable exception: java.nio.file.NoSuchFileException: spark-internal
                

Not using Mixmax yet?  




Sent with Mixmax
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message