Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5CE4318227 for ; Wed, 29 Jul 2015 16:24:00 +0000 (UTC) Received: (qmail 68008 invoked by uid 500); 29 Jul 2015 16:23:58 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 67936 invoked by uid 500); 29 Jul 2015 16:23:58 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 67926 invoked by uid 99); 29 Jul 2015 16:23:58 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jul 2015 16:23:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 15CA7D8998 for ; Wed, 29 Jul 2015 16:23:58 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.001 X-Spam-Level: **** X-Spam-Status: No, score=4.001 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id ozRJxekGTPKv for ; Wed, 29 Jul 2015 16:23:44 +0000 (UTC) Received: from mail-qg0-f42.google.com (mail-qg0-f42.google.com [209.85.192.42]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id A677A2134E for ; Wed, 29 Jul 2015 16:23:43 +0000 (UTC) Received: by qgeh16 with SMTP id h16so6836445qge.3 for ; Wed, 29 Jul 2015 09:23:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=WDvzqOYmaheLMBLpoGmCnGBapaMKUqw2ufsveTz+efo=; b=Cya9M+LtUEELyFq1CEqosSaBVM8gsppDnmLgsYgEfD0AL1oAzq1KgzcIb9E3uus/py aAqsus2iAyxOrYEgL256pdYFO94FfZv9EmOh5hUDPKvXnVx28zRdQSz/+IkQLoOTLDJf E0lLKl8L8Q1+RSaC60X0VtwboHzjKB7vcmln+RzU0Okprp7US4Yc3Bn746ZPpVdx/cew jNtq0fLcXkqCl9mxks+tCPG1ubifnhZ649wsvNwO/lHREEcYFRB4iJxTISI9tEiumyxP DPMLRdDQ+46SJq3SaXMG89Tt+8iC10xBjIWc4XqgofGIzI/C72FlPcKahtFk8ZNteZqk eMlA== X-Gm-Message-State: ALoCoQkVZPI7hbm7FDzQf8xjtBOMp0sSvAjFkh4gRcZ65hXmbQ7A0mZnszbxx5DxOpSGvV922jrM MIME-Version: 1.0 X-Received: by 10.140.31.74 with SMTP id e68mr61551206qge.95.1438187017133; Wed, 29 Jul 2015 09:23:37 -0700 (PDT) Received: by 10.96.154.37 with HTTP; Wed, 29 Jul 2015 09:23:37 -0700 (PDT) In-Reply-To: References: Date: Wed, 29 Jul 2015 18:23:37 +0200 Message-ID: Subject: Re: Computation timeout From: =?UTF-8?B?TG/Dr2MgQ2hhbmVs?= To: user@hive.apache.org Content-Type: multipart/alternative; boundary=001a113a9aee470085051c06021c --001a113a9aee470085051c06021c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I don't think your solution works, as after more than 4 minutes I could still see logs of my job showing that it was running. Do you have a way to check that even if the job was running, it was not being killed by Hive ? Or another solution ? Thanks for your help, Lo=C3=AFc Lo=C3=AFc CHANEL Engineering student at TELECOM Nancy Trainee at Worldline - Villeurbanne 2015-07-29 16:26 GMT+02:00 Lo=C3=AFc Chanel : > Yes, I set it to negative 60. > > It's not a problem if the session is killed. That's actually what I try t= o > do, because I can't allow to a user to try to end an infinite request. > Therefore I'll try your solution :) > > Thanks, > > > Lo=C3=AFc > > Lo=C3=AFc CHANEL > Engineering student at TELECOM Nancy > Trainee at Worldline - Villeurbanne > > 2015-07-29 16:14 GMT+02:00 Xuefu Zhang : > >> Okay. To confirm, you set it to negative 60s? >> >> The next thing you can try is to set >> hive.server2.idle.session.timeou=3D60000 (60sec) and >> hive.server2.idle.session.check.operation=3Dfalse. I'm pretty sure this >> works, but the user's session will be killed though. >> >> --Xuefu >> >> On Wed, Jul 29, 2015 at 7:02 AM, Lo=C3=AFc Chanel < >> loic.chanel@telecomnancy.net> wrote: >> >>> I confirm : I just tried hive.server2.idle.operation.timeout setting it >>> to -60 (seconds), but my veeeeeery slow job have not been killed. The i= ssue >>> here is "what if another user come and try to submit a MapReduce job bu= t >>> the cluster is stuck in an infinite loop ?". >>> >>> Do you or anyone else have another idea ? >>> Thanks, >>> >>> >>> Lo=C3=AFc >>> >>> Lo=C3=AFc CHANEL >>> Engineering student at TELECOM Nancy >>> Trainee at Worldline - Villeurbanne >>> >>> 2015-07-29 15:34 GMT+02:00 Lo=C3=AFc Chanel : >>> >>>> No, because I thought the idea of infinite operation was not very >>>> compatible with the "idle" word (as the operation will not stop runnin= g), >>>> but I'll try :-) >>>> Thanks for the idea, >>>> >>>> >>>> Lo=C3=AFc >>>> >>>> Lo=C3=AFc CHANEL >>>> Engineering student at TELECOM Nancy >>>> Trainee at Worldline - Villeurbanne >>>> >>>> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang : >>>> >>>>> Have you tried hive.server2.idle.operation.timeout? >>>>> >>>>> --Xuefu >>>>> >>>>> On Wed, Jul 29, 2015 at 5:52 AM, Lo=C3=AFc Chanel < >>>>> loic.chanel@telecomnancy.net> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> As I'm trying to build a secured and multi-tenant Hadoop cluster wit= h >>>>>> Hive, I am desperately trying to set a timeout to Hive requests. >>>>>> My idea is that some users can make mistakes such as a join with >>>>>> wrong keys, and therefore start an infinite loop believing that they= are >>>>>> just launching a very heavy job. Therefore, I'd like to set a limit = to the >>>>>> time a request should take, in order to kill the job automatically i= f it >>>>>> exceeds it. >>>>>> >>>>>> As such a notion cannot be set directly in YARN, I saw that >>>>>> MapReduce2 provides with its own native timeout property, and I woul= d like >>>>>> to know if Hive provides with the same property someway. >>>>>> >>>>>> Did anyone heard about such a thing ? >>>>>> >>>>>> Thanks in advance for your help, >>>>>> >>>>>> >>>>>> Lo=C3=AFc >>>>>> >>>>>> Lo=C3=AFc CHANEL >>>>>> Engineering student at TELECOM Nancy >>>>>> Trainee at Worldline - Villeurbanne >>>>>> >>>>> >>>>> >>>> >>> >> > --001a113a9aee470085051c06021c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I don't think your solution works,= as after more than 4 minutes I could still see logs of my job showing that= it was running.
Do you have a way to check that even if the job w= as running, it was not being killed by Hive ?
Or another solution = ?

Thanks for your help,


Lo=C3=AFc

Lo=C3=AFc CHANEL
Enginee= ring student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
<= /div>

2015-07-29 16:26 GMT+02:00 Lo=C3=AFc Chanel = <loic.chanel@telecomnancy.net>:
Yes, I set it to nega= tive 60.

It's not a problem if the session is killed. That= 's actually what I try to do, because I can't allow to a user to tr= y to end an infinite request.
Therefore I'll try your solution= :)

Thanks,


Lo=C3=AFc

=
Lo=C3=AFc CHANEL
Engineering student at TELEC= OM Nancy
Trainee at Worldline - Villeurbanne

2015-07-29 16:= 14 GMT+02:00 Xuefu Zhang <xzhang@cloudera.com>:
Okay. To confirm, you set it = to negative 60s?

The next thing you can try is to set hive.server2.= idle.session.timeou=3D60000 (60sec) and hive.server2.idle.session.check.ope= ration=3Dfalse. I'm pretty sure this works, but the user's session = will be killed though.

<= /div>--Xuefu

On Wed, Jul 29, = 2015 at 7:02 AM, Lo=C3=AFc Chanel <loic.chanel@telecomnancy.net= > wrote:
<= div>
I confirm : I just tried hive.server2.idle.operation.timeout = setting it to -60 (seconds), but my veeeeeery slow job have not been killed= . The issue here is "what if another user come and try to submit a Map= Reduce job but the cluster is stuck in an infinite loop ?".

Do you or anyone else have another idea ?
Thanks,


Lo=C3=AFc

<= div>
Lo=C3=AFc CHANEL
En= gineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne

2015-07-29 15:34 GMT+02:00 = Lo=C3=AFc Chanel <loic.chanel@telecomnancy.net>:<= br>
No, because I thoug= ht the idea of infinite operation was not very compatible with the "id= le" word (as the operation will not stop running), but I'll try :-= )
Thanks for the idea,


Lo=C3=AFc

Lo=C3=AFc CHANEL
Engineering stud= ent at TELECOM Nancy
Trainee at Worldline - Villeurbanne

2015-07-29 15:27 GMT+02:00 = Xuefu Zhang <xzhang@cloudera.com>:
Have you tried hive.server2.idle.operation= .timeout?

--Xuefu

On Wed, Jul 29, 2015 at 5:5= 2 AM, Lo=C3=AFc Chanel <loic.chanel@telecomnancy.net> wrote:
Hi all,

As I'm trying to build a secured= and multi-tenant Hadoop cluster with Hive, I am desperately trying to set = a timeout to Hive requests.
My idea is that some users can make mi= stakes such as a join with wrong keys, and therefore start an infinite loop= believing that they are just launching a very heavy job. Therefore, I'= d like to set a limit to the time a request should take, in order to kill t= he job automatically if it exceeds it.

As such a notion cannot= be set directly in YARN, I saw that MapReduce2 provides with its own nativ= e timeout property, and I would like to know if Hive provides with the same= property someway.

Did anyone heard about such a thing ?
Thanks in advance for your help,


Lo=C3=AFc

<= div>
Lo=C3=AFc CHANEL<= br>Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurba= nne






--001a113a9aee470085051c06021c--