livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Partridge, Lucas (GE Aviation)" <Lucas.Partri...@ge.com>
Subject How to access Python script's stdout when running in Yarn cluster mode?
Date Mon, 27 Nov 2017 12:04:06 GMT
Thanks Jerry.

“Currently there's no Livy api for you to get application log via REST API.”
- That’s a real shame. Are there any plans to address this?  Because this really limits
the usefulness of Livy when I’m using Yarn cluster mode.  It seems I have to make a choice
between running my client in a scalable manner (yarn cluster) or being able to see the logs
programmatically (yarn client), but not both.  My client is multi-threaded; I don’t want
it to have to host multiple concurrent Spark driver applications but it looks like I might
have no choice about this.

Thanks, Lucas.

From: Saisai Shao [mailto:sai.sai.shao@gmail.com]
Sent: 27 November 2017 02:19
To: user@livy.incubator.apache.org
Subject: EXT: Re: How to access Python script's stdout when running in Yarn cluster mode?

Since you're running with yarn cluster mode, the output from your python script should be
part of your yarn application log. you can get it via yarn command like yarn log -applicationId
<applictionId>, or others like Yarn UI. Currently there's no Livy api for you to get
application log via REST API.

Thanks
Jerry

2017-11-24 20:27 GMT+08:00 Partridge, Lucas (GE Aviation) <Lucas.Partridge@ge.com<mailto:Lucas.Partridge@ge.com>>:
Hi,

I’m using Livy’s GET /batches/{batchId}/log method to fetch the log lines from a Python
script I’m running on Spark in Yarn cluster mode. Unfortunately the stdout from my Python
script is not included in the log lines returned by GET /batches/{batchId}/log!

Is this by design, or an unfortunate by-product of running in Yarn cluster mode?

If this is intentional does anyone know how I can access the stdout from my Python script
via Livy please? Preferably without having to change my REST client (a Java app) to use Yarn’s
client deployment mode.

Thanks,
Lucas.


Mime
View raw message