livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Bozarth" <ajboz...@us.ibm.com>
Subject Re: POST /batches failing with "Only local python files are supported"
Date Fri, 03 Nov 2017 20:11:05 GMT
Michael's solution might do it for you, but if it doesn't I would try the
pyFiles arg instead, I haven't used it but it's purpose may be to fix this
exact issue. I can research it more if needed.

                                                                                   
 Alex Bozarth                                                                      
 Software Engineer                                                                 
 Spark Technology Center                                                           
                                                                                   
                                                                                     
                                                                                     
                                                                                     
 E-mail: ajbozart@us.ibm.com                                                         
 GitHub: github.com/ajbozarth                                                        
                                                                   505 Howard Street 
                                                             San Francisco, CA 94105 
                                                                       United States 
                                                                                     








From:	Michael Rhee <mrhee@gogoair.com>
To:	"user@livy.incubator.apache.org"
            <user@livy.incubator.apache.org>
Date:	11/03/2017 07:41 AM
Subject:	Re: POST /batches failing with "Only local python files are
            supported"



Hi Lucas,

I believe you need to modify your Livy configuration file to allow access
to a local directory on your master node. Something like the following:

livy.file.local-dir-whitelist =/home/hadoop-user/

Then use the file:/home/hadoop-user/file argument when you pass your
request to Livy. Hope that helps.

Best,
Michael

From: "Partridge, Lucas (GE Aviation)" <Lucas.Partridge@ge.com>
Reply-To: "user@livy.incubator.apache.org" <user@livy.incubator.apache.org>
Date: Friday, November 3, 2017 at 5:34 AM
To: "user@livy.incubator.apache.org" <user@livy.incubator.apache.org>
Subject: POST /batches failing with "Only local python files are supported"

Thanks for the suggestion Alex. However whenever I try anything beginning
with file:/, file:// or file:/// or file://NNPRDHA/ I get this error:

org.springframework.web.client.RestClientException: Error running rest
call; nested exception is
org.springframework.web.client.HttpClientErrorException: 400 Bad Request

It sounds like from what you’ve said it’s a Spark error rather than a Livy
error, which I didn’t realise before. But whenever I put in a file argument
value without a file: in front it - Livy or Spark? - assumes it’s an HDFS
path and prepends it with hdfs://. Then Spark complains that only local
Python files are supported.

I’ve also tried copying the Python file from hdfs to the local file system
of the Spark Master node. But I can’t specify that path in my POST call
because if I use file: I get 400 Bad Request; and if I don’t use file: it
(Spark?) assumes it’s in hdfs!  Should I use the files argument for
POST /batches too? Or the pyFiles argument, although I assumed that was for
Python libraries required by the main application.  I’ve tried lots of
combinations but none have worked so far.


From: Alex Bozarth [mailto:ajbozart@us.ibm.com]
Sent: 02 November 2017 21:34
To: user@livy.incubator.apache.org
Subject: EXT: Re: POST /batches failing with "Only local python files are
supported"



It sounds like you're passing in a local past and it's being treated as an
HDFS path. Have you tired passing the path in with file:// at the front
(similar to hdfs://) that tells hdfs that the path is local, I've run into
this issue with Spark before.




                                                                                   
 Alex Bozarth                                                                      
 Software Engineer                                                                 
 Spark Technology Center                                                           
                                                                                   





                                                                                    
                                                                                    
                                                                                    
 E-mail: ajbozart@us.ibm.com                                                        
 GitHub: github.com/ajbozarth                                                       
                                                                  505 Howard Street 
                                                            San Francisco, CA 94105 
                                                                      United States 
                                                                                    







nactive hide details for "Partridge, Lucas (GE Aviation)" ---11/02/2017 0
"Partridge, Lucas (GE Aviation)" ---11/02/2017 07:14:58 AM---I want to use
Livy (0.3) to run a Python file that I've placed in HDFS. I'm invoking
POST /batches

From: "Partridge, Lucas (GE Aviation)" <Lucas.Partridge@ge.com>
To: "user@livy.incubator.apache.org" <user@livy.incubator.apache.org>
Date: 11/02/2017 07:14 AM
Subject: POST /batches failing with "Only local python files are supported"




I want to use Livy (0.3) to run a Python file that I’ve placed in HDFS. I’m
invoking POST /batches from a Java REST client, passing in the path to the
HDFS file as the ‘file’ argument of the POST request’s body (
https://github.com/apache/incubator-livy/tree/branch-0.3#request-body-2).
The value I’m providing for ‘file’ is "/user/MyUserName/hello.py".

The POST response says the batch is in state ‘starting’ but when I query it
using GET /batches/{batchId} I see this:

Error: Only local python files are supported:
hdfs://NNPRDHA/user/MyUserName/hello.py

You can see the value has been altered from what I provided. How do I
successfully invoke the Python file please?

(I see someone raised a similar problem at
https://groups.google.com/a/cloudera.org/d/msg/livy-user/6AZeqtVwipg/U46tUjqNBwAJ
 but it’s not clear if or how they solved it.)

Thanks, Lucas.



Mime
View raw message