Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 20EDE200CCA for ; Wed, 5 Jul 2017 04:36:55 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 1FE7B162323; Wed, 5 Jul 2017 02:36:55 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 14925162321 for ; Wed, 5 Jul 2017 04:36:53 +0200 (CEST) Received: (qmail 24763 invoked by uid 500); 5 Jul 2017 02:36:53 -0000 Mailing-List: contact user-help@livy.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@livy.incubator.apache.org Delivered-To: mailing list user@livy.incubator.apache.org Received: (qmail 24753 invoked by uid 99); 5 Jul 2017 02:36:53 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jul 2017 02:36:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id BA7DD1A0501 for ; Wed, 5 Jul 2017 02:36:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.881 X-Spam-Level: * X-Spam-Status: No, score=1.881 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_FONT_LOW_CONTRAST=0.001, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id pVv_tdS4CIvI for ; Wed, 5 Jul 2017 02:36:50 +0000 (UTC) Received: from mail-oi0-f41.google.com (mail-oi0-f41.google.com [209.85.218.41]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 8EC2F5F6C4 for ; Wed, 5 Jul 2017 02:36:50 +0000 (UTC) Received: by mail-oi0-f41.google.com with SMTP id l130so115897385oib.1 for ; Tue, 04 Jul 2017 19:36:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=PkmIceFl76SRmVxehN2y6l83ZnF0o991rP0T4c8JE9I=; b=W8O0fheqySrlGBKkrEmaPwHD9cqI1fJr5LrQKFWquWaYEAaB1KgvT5TzEshtCjQQxo wRE+Tlev9x1vlvoUXYXpXgAIl57P9DNSbPfqrjD31AhX6/A4PpYEuoF3KtlFKq49QnG4 Cqerr1p/YI7pjix18p2poCh7lD3GzPB9IUM6t14utoSg1621Y+ff53aamfx0baCQNBif dkql4q9NY/3L0ARs2La9+soQoO8KmUr4YCac35CSdnoNnhNxZQWTlisZj4iEP54/sLU4 sKDZFsLimL5pjGOz1cwac6szGy+zRDqwziMjDaad2fYNOLlHDlOPF5M5XhYYdaFt/dkq EQJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=PkmIceFl76SRmVxehN2y6l83ZnF0o991rP0T4c8JE9I=; b=rL6empswKY4/1aV7B7zoEIBYlwYHM5ZtcNOxUCi8KASCyYrC96IGYdgKgG5g0HaxYj y6rTUiLT5E0Mkyixn/+OF4n3ZHAr/CsuaIyQO5jeyWB68C7k4rd8KTdoPXa+Bzzcnrg7 P3YIu2j64PC3QA74pO0/MxQxZqF8vSomQliDwu/6TDTfGcJIfU+pkQ/VWx4PFVGhY1S8 60IQJ18O4OrqAX0AuPCFX1r3uTSAeTFVVKi/WelhkuqcQnHPzqf6kPSgc9kqVV6JY7ME VU0fSxkZ+QwBlsFtd8CoI9Jka+p5SAK8nrB7CHghKORi3r3jIu87RcZehEHt2L+V7ydu JqwA== X-Gm-Message-State: AKS2vOwiODexBSeaasWgmbMt7u+NI2totCk6r4kjF3kPo4eiJw2U6r5E gRVWi1pWBXsaL5juuUH1oQ10SR09lEUL X-Received: by 10.202.80.66 with SMTP id e63mr27534833oib.152.1499222203797; Tue, 04 Jul 2017 19:36:43 -0700 (PDT) MIME-Version: 1.0 Received: by 10.182.20.163 with HTTP; Tue, 4 Jul 2017 19:36:43 -0700 (PDT) In-Reply-To: References: From: Saisai Shao Date: Wed, 5 Jul 2017 10:36:43 +0800 Message-ID: Subject: Re: Livy Automatically Deletes Log Files To: user@livy.incubator.apache.org Content-Type: multipart/alternative; boundary="001a113d6a74e59749055388dece" archived-at: Wed, 05 Jul 2017 02:36:55 -0000 --001a113d6a74e59749055388dece Content-Type: text/plain; charset="UTF-8" IIUC, you want to get batch session state even when application was finished long ago, so you're trying to read the state from recovery log. I think this is not a proper way, anyway session metadata and session recovery log will be deleted after timeout. If you want to extend this expiry date, you could set "livy.server.session.state-retain.sec" to a large value, by default it is 600 seconds. On Wed, Jul 5, 2017 at 4:09 AM, Dimosthenis Masouros < demo.masouros@gmail.com> wrote: > Good evening, > > Before I state my problem, for reference reasons, my livy.conf file looks > like this: > > livy.spark.master = yarn > livy.spark.deployMode = cluster > livy.server.recovery.mode = recovery > livy.server.recovery.state-store = filesystem > livy.server.recovery.state-store.url = file:///root/hdfs/livy > livy.server.request-log-retain.days = 5 > > I have set up a livy server on a hadoop namenode. When i submit a new job > everything operates normally and the job is submitted successfully. > > curl -X POST --data '{"file": "/pi.py", "args": ["10"], "name": "test > livy"}' -H "Content-Type: application/json" namenode:8998/batches > > After the job is submitted i am able to see the status of the application > by running: > > curl -X POST namenode:8998/batches/id > > which returns a json file with the id of the job etc. If i list the files > of the /root/hdfs/livy folder on the namenode, during the job's execution I > am able to see the exact same file as the replied from the curl request. > However, the problem i am experiencing is that when the job finishes its > execution, after some period of time (~10-15 mins) the log file of the job > is deleted automatically from the host. Is there any way i can keep this > log file? Or alternatively, is there any way to find an application's > status submitted with livy, using a curl request (or anything familiar)? > > > Thanks for your support. > --001a113d6a74e59749055388dece Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
IIUC, you want to get batch session state even when applic= ation was finished long ago, so you're trying to read the state from re= covery log. I think this is not a proper way, anyway session metadata and s= ession recovery log will be deleted after timeout. If you want to extend th= is expiry date, you could set "livy.server= .session.state-retain.sec" to a large value, by default it is 6= 00 seconds.

On Wed, Jul 5, 2017 at 4:09 AM, Dimosthenis Masouros = <demo.masou= ros@gmail.com> wrote:
Good evening,

Before I state my problem, for refer= ence reasons, my livy.conf file looks like this:

livy.spark.master =3D yarn
livy.spark.deployMode =3D cluster
livy.server.recovery.mode =3D recovery
livy.server.recovery.state-sto= re =3D filesystem
liv= y.server.recovery.state-store.url =3D file:///root/hdfs/livy
livy.server.request-log-retain= .days =3D 5

I have set up a livy server = on a hadoop namenode. When i submit a new job everything operates normally = and the job is submitted successfully.
<= div style=3D"margin:0px;padding:0px;border:0px">
curl -X POST --data '{"file": "/pi.py", "a= rgs": ["10"], "name": "test livy"}' = -H "Content-Type: application/json" namenode:8998/batches<= br>

After the job is submitted i am able to see the status= of the application by running:

<= span style=3D"margin:0px;padding:0px;border:0px;color:rgba(0,0,0,0.87);font= -family:Consolas,"Liberation Mono",Menlo,Courier,monospace;font-s= ize:13.6px;background-color:rgba(0,0,0,0.04)">curl -X POST=C2=A0namenode:8998/batches/id<= br>

=
which returns a json file = with the id of the job etc. If i list the files of the /root/hdfs/livy fold= er on the namenode, during the job's execution I am able to see the exa= ct same file as the replied from the curl request. However, the problem i a= m experiencing is that when the job finishes its execution, after some peri= od of time (~10-15 mins) the log file of the job is deleted automatically f= rom the host. Is there any way i can keep this log file? Or alternatively, = is there any way to find an application's status submitted with livy, u= sing a curl request (or anything familiar)?=C2=A0


=
Thanks for your supp= ort.
=
=

--001a113d6a74e59749055388dece--