hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xuefu Zhang <xzh...@cloudera.com>
Subject Re: Computation timeout
Date Wed, 29 Jul 2015 18:15:53 GMT
this works for me:
In hive-site.xml:
  1. hive.server2.session.check.interva=3000;
  2. hive.server2.idle.operation.timeou=-30000;
restart HiveServer2.

at beeline, I do "analyze table X compute statistics for columns", which
takes longer than 30s. it was aborted by HS2 because of above settings. I
guess it didn't work for you because you didn't have #1.

--Xuefu

On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel <loic.chanel@telecomnancy.net>
wrote:

> I don't think your solution works, as after more than 4 minutes I could
> still see logs of my job showing that it was running.
> Do you have a way to check that even if the job was running, it was not
> being killed by Hive ?
> Or another solution ?
>
> Thanks for your help,
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-07-29 16:26 GMT+02:00 Loïc Chanel <loic.chanel@telecomnancy.net>:
>
>> Yes, I set it to negative 60.
>>
>> It's not a problem if the session is killed. That's actually what I try
>> to do, because I can't allow to a user to try to end an infinite request.
>> Therefore I'll try your solution :)
>>
>> Thanks,
>>
>>
>> Loïc
>>
>> Loïc CHANEL
>> Engineering student at TELECOM Nancy
>> Trainee at Worldline - Villeurbanne
>>
>> 2015-07-29 16:14 GMT+02:00 Xuefu Zhang <xzhang@cloudera.com>:
>>
>>> Okay. To confirm, you set it to negative 60s?
>>>
>>> The next thing you can try is to set
>>> hive.server2.idle.session.timeou=60000 (60sec) and
>>> hive.server2.idle.session.check.operation=false. I'm pretty sure this
>>> works, but the user's session will be killed though.
>>>
>>> --Xuefu
>>>
>>> On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel <
>>> loic.chanel@telecomnancy.net> wrote:
>>>
>>>> I confirm : I just tried hive.server2.idle.operation.timeout setting it
>>>> to -60 (seconds), but my veeeeeery slow job have not been killed. The issue
>>>> here is "what if another user come and try to submit a MapReduce job but
>>>> the cluster is stuck in an infinite loop ?".
>>>>
>>>> Do you or anyone else have another idea ?
>>>> Thanks,
>>>>
>>>>
>>>> Loïc
>>>>
>>>> Loïc CHANEL
>>>> Engineering student at TELECOM Nancy
>>>> Trainee at Worldline - Villeurbanne
>>>>
>>>> 2015-07-29 15:34 GMT+02:00 Loïc Chanel <loic.chanel@telecomnancy.net>:
>>>>
>>>>> No, because I thought the idea of infinite operation was not very
>>>>> compatible with the "idle" word (as the operation will not stop running),
>>>>> but I'll try :-)
>>>>> Thanks for the idea,
>>>>>
>>>>>
>>>>> Loïc
>>>>>
>>>>> Loïc CHANEL
>>>>> Engineering student at TELECOM Nancy
>>>>> Trainee at Worldline - Villeurbanne
>>>>>
>>>>> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang <xzhang@cloudera.com>:
>>>>>
>>>>>> Have you tried hive.server2.idle.operation.timeout?
>>>>>>
>>>>>> --Xuefu
>>>>>>
>>>>>> On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <
>>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> As I'm trying to build a secured and multi-tenant Hadoop cluster
>>>>>>> with Hive, I am desperately trying to set a timeout to Hive requests.
>>>>>>> My idea is that some users can make mistakes such as a join with
>>>>>>> wrong keys, and therefore start an infinite loop believing that
they are
>>>>>>> just launching a very heavy job. Therefore, I'd like to set a
limit to the
>>>>>>> time a request should take, in order to kill the job automatically
if it
>>>>>>> exceeds it.
>>>>>>>
>>>>>>> As such a notion cannot be set directly in YARN, I saw that
>>>>>>> MapReduce2 provides with its own native timeout property, and
I would like
>>>>>>> to know if Hive provides with the same property someway.
>>>>>>>
>>>>>>> Did anyone heard about such a thing ?
>>>>>>>
>>>>>>> Thanks in advance for your help,
>>>>>>>
>>>>>>>
>>>>>>> Loïc
>>>>>>>
>>>>>>> Loïc CHANEL
>>>>>>> Engineering student at TELECOM Nancy
>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message