livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saisai Shao <>
Subject Re: Livy Automatically Deletes Log Files
Date Wed, 05 Jul 2017 02:36:43 GMT
IIUC, you want to get batch session state even when application was
finished long ago, so you're trying to read the state from recovery log. I
think this is not a proper way, anyway session metadata and session
recovery log will be deleted after timeout. If you want to extend this
expiry date, you could set "livy.server.session.state-retain.sec" to a
large value, by default it is 600 seconds.

On Wed, Jul 5, 2017 at 4:09 AM, Dimosthenis Masouros <> wrote:

> Good evening,
> Before I state my problem, for reference reasons, my livy.conf file looks
> like this:
> livy.spark.master = yarn
> livy.spark.deployMode = cluster
> livy.server.recovery.mode = recovery
> livy.server.recovery.state-store = filesystem
> livy.server.recovery.state-store.url = file:///root/hdfs/livy
> livy.server.request-log-retain.days = 5
> I have set up a livy server on a hadoop namenode. When i submit a new job
> everything operates normally and the job is submitted successfully.
> curl -X POST --data '{"file": "/", "args": ["10"], "name": "test
> livy"}' -H "Content-Type: application/json" namenode:8998/batches
> After the job is submitted i am able to see the status of the application
> by running:
> curl -X POST namenode:8998/batches/id
> which returns a json file with the id of the job etc. If i list the files
> of the /root/hdfs/livy folder on the namenode, during the job's execution I
> am able to see the exact same file as the replied from the curl request.
> However, the problem i am experiencing is that when the job finishes its
> execution, after some period of time (~10-15 mins) the log file of the job
> is deleted automatically from the host. Is there any way i can keep this
> log file? Or alternatively, is there any way to find an application's
> status submitted with livy, using a curl request (or anything familiar)?
> Thanks for your support.

View raw message