livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Rhee <>
Subject Re: POST /batches failing with "Only local python files are supported"
Date Fri, 03 Nov 2017 14:41:19 GMT
Hi Lucas,

I believe you need to modify your Livy configuration file to allow access to a local directory
on your master node. Something like the following:

livy.file.local-dir-whitelist =/home/hadoop-user/

Then use the file:/home/hadoop-user/file argument when you pass your request to Livy. Hope
that helps.


From: "Partridge, Lucas (GE Aviation)" <>
Reply-To: "" <>
Date: Friday, November 3, 2017 at 5:34 AM
To: "" <>
Subject: POST /batches failing with "Only local python files are supported"

Thanks for the suggestion Alex. However whenever I try anything beginning with file:/, file://
or file:/// or file://NNPRDHA/<file:///NNPRDHA/> I get this error:

org.springframework.web.client.RestClientException: Error running rest call; nested exception
is org.springframework.web.client.HttpClientErrorException: 400 Bad Request

It sounds like from what you’ve said it’s a Spark error rather than a Livy error, which
I didn’t realise before. But whenever I put in a file argument value without a file: in
front it - Livy or Spark? - assumes it’s an HDFS path and prepends it with hdfs://. Then
Spark complains that only local Python files are supported.

I’ve also tried copying the Python file from hdfs to the local file system of the Spark
Master node. But I can’t specify that path in my POST call because if I use file: I get
400 Bad Request; and if I don’t use file: it (Spark?) assumes it’s in hdfs!  Should I
use the files argument for POST /batches too? Or the pyFiles argument, although I assumed
that was for Python libraries required by the main application.  I’ve tried lots of combinations
but none have worked so far.

From: Alex Bozarth []
Sent: 02 November 2017 21:34
Subject: EXT: Re: POST /batches failing with "Only local python files are supported"

It sounds like you're passing in a local past and it's being treated as an HDFS path. Have
you tired passing the path in with file:// at the front (similar to hdfs://) that tells hdfs
that the path is local, I've run into this issue with Spark before.

Alex Bozarth
Software Engineer
Spark Technology Center



505 Howard Street
San Francisco, CA 94105
United States

[nactive hide details for "Partridge, Lucas (GE Aviation)" ---11/02/2017 0]"Partridge, Lucas
(GE Aviation)" ---11/02/2017 07:14:58 AM---I want to use Livy (0.3) to run a Python file that
I've placed in HDFS. I'm invoking POST /batches

From: "Partridge, Lucas (GE Aviation)" <<>>
To: "<>" <<>>
Date: 11/02/2017 07:14 AM
Subject: POST /batches failing with "Only local python files are supported"


I want to use Livy (0.3) to run a Python file that I’ve placed in HDFS. I’m invoking POST
/batches from a Java REST client, passing in the path to the HDFS file as the ‘file’ argument
of the POST request’s body (<>).
The value I’m providing for ‘file’ is "/user/MyUserName/".

The POST response says the batch is in state ‘starting’ but when I query it using GET
/batches/{batchId} I see this:

Error: Only local python files are supported: hdfs://NNPRDHA/user/MyUserName/

You can see the value has been altered from what I provided. How do I successfully invoke
the Python file please?

(I see someone raised a similar problem at<>
but it’s not clear if or how they solved it.)

Thanks, Lucas.

View raw message