livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Partridge, Lucas (GE Aviation)" <Lucas.Partri...@ge.com>
Subject RE: POST /batches failing with "Only local python files are supported"
Date Tue, 07 Nov 2017 13:42:06 GMT
Michael was correct - thank you! Once my directory had been added to livy.file.local-dir-whitelist
the POST /batches request worked using just file:/foo/bar/hello.py for the file argument.
No pyFiles arg was required.

One subtlety was that the Livy server was not running on the same machine as the Spark master
node. And it appears the local directory has to be a directory on the Livy server itself.
 If I just put the file on the Spark master node I got:
python: can't open file '/foo/bar/hello.py': [Errno 2] No such file or directory
java.lang.Exception: spark-submit exited with code 2}.

So is this true of all the file-related args in the POST /batches call then? i.e., file, jars,
pyFiles and files must all reference paths on the Livy server itself? I’m currently using
Livy 0.3.

Thanks, Lucas.

From: Partridge, Lucas (GE Aviation)
Sent: 06 November 2017 16:31
To: user@livy.incubator.apache.org
Subject: EXT: POST /batches failing with "Only local python files are supported"

Michael – thanks for your suggestion. At the moment any request I make with a file:/ argument
immediately gets rejected as a ‘400 Bad Request’ but I’ve asked for livy.file.local-dir-whitelist
to be modified in case that’ll make a difference.  Unfortunately I don’t administer the
Livy server so I’ll have to wait on that…

Alex – I tried the pyFiles arg with both the hdfs: and file: path but it made no difference
to the “Only local python files are supported” error. Feel free to research it more but
I would expect the purpose of the pyFiles argument to be to supply any missing Python libraries
that the Python application supplied in the file arg may depend on.

Thanks, Lucas.

From: Alex Bozarth [mailto:ajbozart@us.ibm.com]
Sent: 03 November 2017 20:11
To: user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>
Subject: EXT: Re: POST /batches failing with "Only local python files are supported"


Michael's solution might do it for you, but if it doesn't I would try the pyFiles arg instead,
I haven't used it but it's purpose may be to fix this exact issue. I can research it more
if needed.
Alex Bozarth
Software Engineer
Spark Technology Center




________________________________

E-mail: ajbozart@us.ibm.com<mailto:ajbozart@us.ibm.com>
GitHub: github.com/ajbozarth<https://github.com/ajbozarth>



505 Howard Street
San Francisco, CA 94105
United States




[Inactive hide details for Michael Rhee ---11/03/2017 07:41:35 AM---Hi Lucas, I believe you
need to modify your Livy configurati]Michael Rhee ---11/03/2017 07:41:35 AM---Hi Lucas, I
believe you need to modify your Livy configuration file to allow access to a local direc

From: Michael Rhee <mrhee@gogoair.com<mailto:mrhee@gogoair.com>>
To: "user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>" <user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>>
Date: 11/03/2017 07:41 AM
Subject: Re: POST /batches failing with "Only local python files are supported"

________________________________



Hi Lucas,

I believe you need to modify your Livy configuration file to allow access to a local directory
on your master node. Something like the following:

livy.file.local-dir-whitelist =/home/hadoop-user/

Then use the file:/home/hadoop-user/file argument when you pass your request to Livy. Hope
that helps.

Best,
Michael

From: "Partridge, Lucas (GE Aviation)" <Lucas.Partridge@ge.com<mailto:Lucas.Partridge@ge.com>>
Reply-To: "user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>" <user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>>
Date: Friday, November 3, 2017 at 5:34 AM
To: "user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>" <user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>>
Subject: POST /batches failing with "Only local python files are supported"

Thanks for the suggestion Alex. However whenever I try anything beginning with file:/, file://
or file:/// or file://NNPRDHA/<file:///NNPRDHA/> I get this error:

org.springframework.web.client.RestClientException: Error running rest call; nested exception
is org.springframework.web.client.HttpClientErrorException: 400 Bad Request

It sounds like from what you’ve said it’s a Spark error rather than a Livy error, which
I didn’t realise before. But whenever I put in a file argument value without a file: in
front it - Livy or Spark? - assumes it’s an HDFS path and prepends it with hdfs://. Then
Spark complains that only local Python files are supported.

I’ve also tried copying the Python file from hdfs to the local file system of the Spark
Master node. But I can’t specify that path in my POST call because if I use file: I get
400 Bad Request; and if I don’t use file: it (Spark?) assumes it’s in hdfs! Should I use
the files argument for POST /batches too? Or the pyFiles argument, although I assumed that
was for Python libraries required by the main application. I’ve tried lots of combinations
but none have worked so far.


From: Alex Bozarth [mailto:ajbozart@us.ibm.com]
Sent: 02 November 2017 21:34
To: user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>
Subject: EXT: Re: POST /batches failing with "Only local python files are supported"

It sounds like you're passing in a local past and it's being treated as an HDFS path. Have
you tired passing the path in with file:// at the front (similar to hdfs://) that tells hdfs
that the path is local, I've run into this issue with Spark before.
Alex Bozarth
Software Engineer
Spark Technology Center




________________________________

E-mail: ajbozart@us.ibm.com<mailto:ajbozart@us.ibm.com>
GitHub: github.com/ajbozarth<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ajbozarth&d=DwMGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo&m=r8pN3Zj5uNwAAQKOlGa68bNHE6RQSAIRPJ7nYhXV76A&s=3ZzlsLWIdaiBmKYONNccTVS0ZH4-fTwkxHEW87QdU9k&e=>



505 Howard Street
San Francisco, CA 94105
United States




[nactive hide details for "Partridge, Lucas (GE Aviation)" ---11/02/2017 0]"Partridge, Lucas
(GE Aviation)" ---11/02/2017 07:14:58 AM---I want to use Livy (0.3) to run a Python file that
I've placed in HDFS. I'm invoking POST /batches

From: "Partridge, Lucas (GE Aviation)" <Lucas.Partridge@ge.com<mailto:Lucas.Partridge@ge.com>>
To: "user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>" <user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>>
Date: 11/02/2017 07:14 AM
Subject: POST /batches failing with "Only local python files are supported"

________________________________




I want to use Livy (0.3) to run a Python file that I’ve placed in HDFS. I’m invoking POST
/batches from a Java REST client, passing in the path to the HDFS file as the ‘file’ argument
of the POST request’s body (https://github.com/apache/incubator-livy/tree/branch-0.3#request-body-2<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dlivy_tree_branch-2D0.3-23request-2Dbody-2D2&d=DwMFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo&m=YIUsDYuvjVmUUxeP7zjxNhpuBW6QdaXW6qnaEbUiGf0&s=6Za80aGx9OBp6GXr8Ze8Di9eqsBXKDD4TraM-8HdumA&e=>).
The value I’m providing for ‘file’ is "/user/MyUserName/hello.py".

The POST response says the batch is in state ‘starting’ but when I query it using GET
/batches/{batchId} I see this:

Error: Only local python files are supported: hdfs://NNPRDHA/user/MyUserName/hello.py

You can see the value has been altered from what I provided. How do I successfully invoke
the Python file please?

(I see someone raised a similar problem at https://groups.google.com/a/cloudera.org/d/msg/livy-user/6AZeqtVwipg/U46tUjqNBwAJ<https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_a_cloudera.org_d_msg_livy-2Duser_6AZeqtVwipg_U46tUjqNBwAJ&d=DwMFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo&m=YIUsDYuvjVmUUxeP7zjxNhpuBW6QdaXW6qnaEbUiGf0&s=FIU8MiykO7n_e8W17PX0vcMILUPvRR8Ga9vYr9DlSW0&e=>
but it’s not clear if or how they solved it.)

Thanks, Lucas.


Mime
View raw message