Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1313018F72 for ; Thu, 30 Jul 2015 07:43:43 +0000 (UTC) Received: (qmail 33963 invoked by uid 500); 30 Jul 2015 07:43:41 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 33891 invoked by uid 500); 30 Jul 2015 07:43:41 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 33881 invoked by uid 99); 30 Jul 2015 07:43:41 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jul 2015 07:43:41 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id B3F97C0723 for ; Thu, 30 Jul 2015 07:43:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.001 X-Spam-Level: *** X-Spam-Status: No, score=3.001 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id GfhIpRP_crnT for ; Thu, 30 Jul 2015 07:43:28 +0000 (UTC) Received: from mail-qk0-f179.google.com (mail-qk0-f179.google.com [209.85.220.179]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 39BC8272A6 for ; Thu, 30 Jul 2015 07:43:28 +0000 (UTC) Received: by qkfc129 with SMTP id c129so15333985qkf.1 for ; Thu, 30 Jul 2015 00:41:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=134IZQhSiRXpz4yLrycIa31D1LZQoB617CIAFo3JXMU=; b=iMTp/tjDcPxlw+fOKeMs1u1AZuhbe10PGS6PGsNwsXvD2CLUIQb7WhoZG3U6NKb1xn WVomj8OKd952MFDFZi3QOeHGmXvTUepD/WYsVw0vk7pe9Lk1hNSgIDSs3hcyw+s6TsSq dvCB9pPN1axI3KR+SPmrWt6dO5zqhi9SVLtjVf4vXHH9TMSqLcl8Pgq0bXpDJuANfG3G /njgkSxg+z/B/oGOLuWA1Fqb/pQO9/1zuJYcaHbwt4Hy8lAcxgbP/9Rf3BvLNUVwznPg swbPe4Z1QRu3g208ucSBBtbhD63QdZH8PmALf0QuQFnJq5MW9XiEAZx0O4m01A82ROra b2og== X-Gm-Message-State: ALoCoQknxOYdep7mAxz4S4BcWdU1VjAgJFUqV6s83J+FO1fV5+S9pmKxrvChK2j7C852yobJmKoX MIME-Version: 1.0 X-Received: by 10.55.27.97 with SMTP id b94mr67959084qkb.3.1438242111298; Thu, 30 Jul 2015 00:41:51 -0700 (PDT) Received: by 10.96.154.37 with HTTP; Thu, 30 Jul 2015 00:41:51 -0700 (PDT) In-Reply-To: References: Date: Thu, 30 Jul 2015 09:41:51 +0200 Message-ID: Subject: Re: Computation timeout From: =?UTF-8?B?TG/Dr2MgQ2hhbmVs?= To: user@hive.apache.org Content-Type: multipart/alternative; boundary=001a1147f09e2543f4051c12d6c8 --001a1147f09e2543f4051c12d6c8 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable My bad, I think I just mixed up the properties. At the end of the day, everything seems to work as you described. Thanks a lot ! Lo=C3=AFc Lo=C3=AFc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-30 9:31 GMT+02:00 Lo=C3=AFc Chanel : > Rats, I think I just figured it out. > #2 Is NEGATIVE 3000, right ? I set it to positive yesterday. > As for #1, I think it is the default value, so I am not sure I have to se= t > it. > > Can you confirm that there is a typo on the name of your properties > (missing last letter) and that is not the actual name of the properties ? > > I'll try again and keep you informed > > > Lo=C3=AFc CHANEL > Engineering student at TELECOM Nancy > Trainee at Worldline - Villeurbanne > > 2015-07-29 20:15 GMT+02:00 Xuefu Zhang : > >> this works for me: >> In hive-site.xml: >> 1. hive.server2.session.check.interva=3D3000; >> 2. hive.server2.idle.operation.timeou=3D-30000; >> restart HiveServer2. >> >> at beeline, I do "analyze table X compute statistics for columns", which >> takes longer than 30s. it was aborted by HS2 because of above settings. = I >> guess it didn't work for you because you didn't have #1. >> >> --Xuefu >> >> On Wed, Jul 29, 2015 at 9:23 AM, Lo=C3=AFc Chanel < >> loic.chanel@telecomnancy.net> wrote: >> >>> I don't think your solution works, as after more than 4 minutes I could >>> still see logs of my job showing that it was running. >>> Do you have a way to check that even if the job was running, it was not >>> being killed by Hive ? >>> Or another solution ? >>> >>> Thanks for your help, >>> >>> >>> Lo=C3=AFc >>> >>> Lo=C3=AFc CHANEL >>> Engineering student at TELECOM Nancy >>> Trainee at Worldline - Villeurbanne >>> >>> 2015-07-29 16:26 GMT+02:00 Lo=C3=AFc Chanel : >>> >>>> Yes, I set it to negative 60. >>>> >>>> It's not a problem if the session is killed. That's actually what I tr= y >>>> to do, because I can't allow to a user to try to end an infinite reque= st. >>>> Therefore I'll try your solution :) >>>> >>>> Thanks, >>>> >>>> >>>> Lo=C3=AFc >>>> >>>> Lo=C3=AFc CHANEL >>>> Engineering student at TELECOM Nancy >>>> Trainee at Worldline - Villeurbanne >>>> >>>> 2015-07-29 16:14 GMT+02:00 Xuefu Zhang : >>>> >>>>> Okay. To confirm, you set it to negative 60s? >>>>> >>>>> The next thing you can try is to set >>>>> hive.server2.idle.session.timeou=3D60000 (60sec) and >>>>> hive.server2.idle.session.check.operation=3Dfalse. I'm pretty sure th= is >>>>> works, but the user's session will be killed though. >>>>> >>>>> --Xuefu >>>>> >>>>> On Wed, Jul 29, 2015 at 7:02 AM, Lo=C3=AFc Chanel < >>>>> loic.chanel@telecomnancy.net> wrote: >>>>> >>>>>> I confirm : I just tried hive.server2.idle.operation.timeout setting >>>>>> it to -60 (seconds), but my veeeeeery slow job have not been killed.= The >>>>>> issue here is "what if another user come and try to submit a MapRedu= ce job >>>>>> but the cluster is stuck in an infinite loop ?". >>>>>> >>>>>> Do you or anyone else have another idea ? >>>>>> Thanks, >>>>>> >>>>>> >>>>>> Lo=C3=AFc >>>>>> >>>>>> Lo=C3=AFc CHANEL >>>>>> Engineering student at TELECOM Nancy >>>>>> Trainee at Worldline - Villeurbanne >>>>>> >>>>>> 2015-07-29 15:34 GMT+02:00 Lo=C3=AFc Chanel >>>>>> : >>>>>> >>>>>>> No, because I thought the idea of infinite operation was not very >>>>>>> compatible with the "idle" word (as the operation will not stop run= ning), >>>>>>> but I'll try :-) >>>>>>> Thanks for the idea, >>>>>>> >>>>>>> >>>>>>> Lo=C3=AFc >>>>>>> >>>>>>> Lo=C3=AFc CHANEL >>>>>>> Engineering student at TELECOM Nancy >>>>>>> Trainee at Worldline - Villeurbanne >>>>>>> >>>>>>> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang : >>>>>>> >>>>>>>> Have you tried hive.server2.idle.operation.timeout? >>>>>>>> >>>>>>>> --Xuefu >>>>>>>> >>>>>>>> On Wed, Jul 29, 2015 at 5:52 AM, Lo=C3=AFc Chanel < >>>>>>>> loic.chanel@telecomnancy.net> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> As I'm trying to build a secured and multi-tenant Hadoop cluster >>>>>>>>> with Hive, I am desperately trying to set a timeout to Hive reque= sts. >>>>>>>>> My idea is that some users can make mistakes such as a join with >>>>>>>>> wrong keys, and therefore start an infinite loop believing that t= hey are >>>>>>>>> just launching a very heavy job. Therefore, I'd like to set a lim= it to the >>>>>>>>> time a request should take, in order to kill the job automaticall= y if it >>>>>>>>> exceeds it. >>>>>>>>> >>>>>>>>> As such a notion cannot be set directly in YARN, I saw that >>>>>>>>> MapReduce2 provides with its own native timeout property, and I w= ould like >>>>>>>>> to know if Hive provides with the same property someway. >>>>>>>>> >>>>>>>>> Did anyone heard about such a thing ? >>>>>>>>> >>>>>>>>> Thanks in advance for your help, >>>>>>>>> >>>>>>>>> >>>>>>>>> Lo=C3=AFc >>>>>>>>> >>>>>>>>> Lo=C3=AFc CHANEL >>>>>>>>> Engineering student at TELECOM Nancy >>>>>>>>> Trainee at Worldline - Villeurbanne >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > --001a1147f09e2543f4051c12d6c8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
My bad, I think I just mixed up the properties.<= br>At the end of the day, everything seems to work as you described.
Thanks a lot !


Lo=C3=AFc



Lo=C3=AFc = CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Vi= lleurbanne

2015-07-30 9:31 GMT+02:00 Lo=C3=AFc Chanel <= span dir=3D"ltr"><loic.chanel@telecomnancy.net>:
Rats, I think I just figure= d it out.
#2 Is NEGATIVE 3000, right ? I set it to positive yester= day.
As for #1, I think it is the default value, so I am not sure = I have to set it.

Can you confirm that there is a typo on= the name of your properties (missing last letter) and that is not the actu= al name of the properties ?

I'll try again and = keep you informed


Lo=C3=AFc CHANEL
Engineering student at TELECOM Nancy=
Trainee at Worldline - Villeurbanne

2015-07-29 20:= 15 GMT+02:00 Xuefu Zhang <xzhang@cloudera.com>:
this works for me:<= br>
In hive-site.xml:
=C2=A0 1. hive.server2.sessi= on.check.interva=3D3000;
=C2=A0 2. hive.= server2.idle.operation.timeou=3D-30000;
restart = HiveServer2.

at beeline, I do "analyze = table X compute statistics for columns", which takes longer than 30s. = it was aborted by HS2 because of above settings. I guess it didn't work= for you because you didn't have #1.
<= /font>

--Xuefu
On Wed, Jul 29, 2015 at 9:23 AM, Lo=C3=AFc Chan= el <loic.chanel@telecomnancy.net> wrote:
I don't = think your solution works, as after more than 4 minutes I could still see l= ogs of my job showing that it was running.
Do you have a way to ch= eck that even if the job was running, it was not being killed by Hive ?
=
Or another solution ?

Thanks for your help,


<= /div>Lo=C3=AFc

=
Lo=C3=AFc CHANEL
E= ngineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne<= /div>

2015-07-29 16:26 GMT+02:00 = Lo=C3=AFc Chanel <loic.chanel@telecomnancy.net>:<= br>
Yes,= I set it to negative 60.

It's not a problem if the sessio= n is killed. That's actually what I try to do, because I can't allo= w to a user to try to end an infinite request.
Therefore I'll = try your solution :)

Thanks,


Lo=C3=AFc

Lo=C3=AFc CHANEL
Engineering student at= TELECOM Nancy
Trainee at Worldline - Villeurbanne

2015-07-29 16:14 GMT+02:00 = Xuefu Zhang <xzhang@cloudera.com>:
Okay. To confirm, you set it to negative 6= 0s?

The next thing you can try is to set hive.server2.idle.session.= timeou=3D60000 (60sec) and hive.server2.idle.session.check.operation=3Dfals= e. I'm pretty sure this works, but the user's session will be kille= d though.

--Xuefu

On Wed, Jul 29, 2015 at 7:0= 2 AM, Lo=C3=AFc Chanel <loic.chanel@telecomnancy.net> wrote:
I confirm : I just tried hive.server2.idle.operation.timeout setting it = to -60 (seconds), but my veeeeeery slow job have not been killed. The issue= here is "what if another user come and try to submit a MapReduce job = but the cluster is stuck in an infinite loop ?".

Do you o= r anyone else have another idea ?
Thanks,


Lo=C3= =AFc

=
Lo=C3=AFc CHANEL
Engineering= student at TELECOM Nancy
Trainee at Worldline - Villeurbanne

2015-07-29 15:34 GMT+02:00 = Lo=C3=AFc Chanel <loic.chanel@telecomnancy.net>:<= br>
No, because I thoug= ht the idea of infinite operation was not very compatible with the "id= le" word (as the operation will not stop running), but I'll try :-= )
Thanks for the idea,


Lo=C3=AFc

Lo=C3=AFc CHANEL
Engineering stud= ent at TELECOM Nancy
Trainee at Worldline - Villeurbanne

2015-07-29 15:27 GMT+02:00 = Xuefu Zhang <xzhang@cloudera.com>:
Have you tried hive.server2.idle.operation= .timeout?

--Xuefu

On Wed, Jul 29, 2015 at 5:5= 2 AM, Lo=C3=AFc Chanel <loic.chanel@telecomnancy.net> wrote:
Hi all,

As I'm trying to build a secured= and multi-tenant Hadoop cluster with Hive, I am desperately trying to set = a timeout to Hive requests.
My idea is that some users can make mi= stakes such as a join with wrong keys, and therefore start an infinite loop= believing that they are just launching a very heavy job. Therefore, I'= d like to set a limit to the time a request should take, in order to kill t= he job automatically if it exceeds it.

As such a notion cannot= be set directly in YARN, I saw that MapReduce2 provides with its own nativ= e timeout property, and I would like to know if Hive provides with the same= property someway.

Did anyone heard about such a thing ?
Thanks in advance for your help,


Lo=C3=AFc

<= div>
Lo=C3=AFc CHANEL<= br>Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurba= nne









--001a1147f09e2543f4051c12d6c8--