livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Rhee <mr...@gogoair.com>
Subject Re: POST /batches failing with "Only local python files are supported"
Date Fri, 03 Nov 2017 14:41:19 GMT
Hi Lucas,

I believe you need to modify your Livy configuration file to allow access to a local directory
on your master node. Something like the following:

livy.file.local-dir-whitelist =/home/hadoop-user/

Then use the file:/home/hadoop-user/file argument when you pass your request to Livy. Hope
that helps.

Best,
Michael

From: "Partridge, Lucas (GE Aviation)" <Lucas.Partridge@ge.com>
Reply-To: "user@livy.incubator.apache.org" <user@livy.incubator.apache.org>
Date: Friday, November 3, 2017 at 5:34 AM
To: "user@livy.incubator.apache.org" <user@livy.incubator.apache.org>
Subject: POST /batches failing with "Only local python files are supported"

Thanks for the suggestion Alex. However whenever I try anything beginning with file:/, file://
or file:/// or file://NNPRDHA/<file:///NNPRDHA/> I get this error:

org.springframework.web.client.RestClientException: Error running rest call; nested exception
is org.springframework.web.client.HttpClientErrorException: 400 Bad Request

It sounds like from what you’ve said it’s a Spark error rather than a Livy error, which
I didn’t realise before. But whenever I put in a file argument value without a file: in
front it - Livy or Spark? - assumes it’s an HDFS path and prepends it with hdfs://. Then
Spark complains that only local Python files are supported.

I’ve also tried copying the Python file from hdfs to the local file system of the Spark
Master node. But I can’t specify that path in my POST call because if I use file: I get
400 Bad Request; and if I don’t use file: it (Spark?) assumes it’s in hdfs!  Should I
use the files argument for POST /batches too? Or the pyFiles argument, although I assumed
that was for Python libraries required by the main application.  I’ve tried lots of combinations
but none have worked so far.


From: Alex Bozarth [mailto:ajbozart@us.ibm.com]
Sent: 02 November 2017 21:34
To: user@livy.incubator.apache.org
Subject: EXT: Re: POST /batches failing with "Only local python files are supported"


It sounds like you're passing in a local past and it's being treated as an HDFS path. Have
you tired passing the path in with file:// at the front (similar to hdfs://) that tells hdfs
that the path is local, I've run into this issue with Spark before.


Alex Bozarth
Software Engineer
Spark Technology Center




________________________________

E-mail: ajbozart@us.ibm.com<mailto:ajbozart@us.ibm.com>
GitHub: github.com/ajbozarth<https://github.com/ajbozarth>



505 Howard Street
San Francisco, CA 94105
United States




[nactive hide details for "Partridge, Lucas (GE Aviation)" ---11/02/2017 0]"Partridge, Lucas
(GE Aviation)" ---11/02/2017 07:14:58 AM---I want to use Livy (0.3) to run a Python file that
I've placed in HDFS. I'm invoking POST /batches

From: "Partridge, Lucas (GE Aviation)" <Lucas.Partridge@ge.com<mailto:Lucas.Partridge@ge.com>>
To: "user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>" <user@livy.incubator.apache.org<mailto:user@livy.incubator.apache.org>>
Date: 11/02/2017 07:14 AM
Subject: POST /batches failing with "Only local python files are supported"

________________________________



I want to use Livy (0.3) to run a Python file that I’ve placed in HDFS. I’m invoking POST
/batches from a Java REST client, passing in the path to the HDFS file as the ‘file’ argument
of the POST request’s body (https://github.com/apache/incubator-livy/tree/branch-0.3#request-body-2<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dlivy_tree_branch-2D0.3-23request-2Dbody-2D2&d=DwMFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo&m=YIUsDYuvjVmUUxeP7zjxNhpuBW6QdaXW6qnaEbUiGf0&s=6Za80aGx9OBp6GXr8Ze8Di9eqsBXKDD4TraM-8HdumA&e=>).
The value I’m providing for ‘file’ is "/user/MyUserName/hello.py".

The POST response says the batch is in state ‘starting’ but when I query it using GET
/batches/{batchId} I see this:

Error: Only local python files are supported: hdfs://NNPRDHA/user/MyUserName/hello.py

You can see the value has been altered from what I provided. How do I successfully invoke
the Python file please?

(I see someone raised a similar problem at https://groups.google.com/a/cloudera.org/d/msg/livy-user/6AZeqtVwipg/U46tUjqNBwAJ<https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_a_cloudera.org_d_msg_livy-2Duser_6AZeqtVwipg_U46tUjqNBwAJ&d=DwMFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=S1_S7Dymu4ZL6g7L21O78VQZ53vEnAyZ-cx37DPYDyo&m=YIUsDYuvjVmUUxeP7zjxNhpuBW6QdaXW6qnaEbUiGf0&s=FIU8MiykO7n_e8W17PX0vcMILUPvRR8Ga9vYr9DlSW0&e=>
but it’s not clear if or how they solved it.)

Thanks, Lucas.


Mime
View raw message