hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Loïc Chanel <loic.cha...@telecomnancy.net>
Subject Re: Computation timeout
Date Thu, 30 Jul 2015 07:41:51 GMT
My bad, I think I just mixed up the properties.
At the end of the day, everything seems to work as you described.

Thanks a lot !


Loïc



Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne

2015-07-30 9:31 GMT+02:00 Loïc Chanel <loic.chanel@telecomnancy.net>:

> Rats, I think I just figured it out.
> #2 Is NEGATIVE 3000, right ? I set it to positive yesterday.
> As for #1, I think it is the default value, so I am not sure I have to set
> it.
>
> Can you confirm that there is a typo on the name of your properties
> (missing last letter) and that is not the actual name of the properties ?
>
> I'll try again and keep you informed
>
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-07-29 20:15 GMT+02:00 Xuefu Zhang <xzhang@cloudera.com>:
>
>> this works for me:
>> In hive-site.xml:
>>   1. hive.server2.session.check.interva=3000;
>>   2. hive.server2.idle.operation.timeou=-30000;
>> restart HiveServer2.
>>
>> at beeline, I do "analyze table X compute statistics for columns", which
>> takes longer than 30s. it was aborted by HS2 because of above settings. I
>> guess it didn't work for you because you didn't have #1.
>>
>> --Xuefu
>>
>> On Wed, Jul 29, 2015 at 9:23 AM, Loïc Chanel <
>> loic.chanel@telecomnancy.net> wrote:
>>
>>> I don't think your solution works, as after more than 4 minutes I could
>>> still see logs of my job showing that it was running.
>>> Do you have a way to check that even if the job was running, it was not
>>> being killed by Hive ?
>>> Or another solution ?
>>>
>>> Thanks for your help,
>>>
>>>
>>> Loïc
>>>
>>> Loïc CHANEL
>>> Engineering student at TELECOM Nancy
>>> Trainee at Worldline - Villeurbanne
>>>
>>> 2015-07-29 16:26 GMT+02:00 Loïc Chanel <loic.chanel@telecomnancy.net>:
>>>
>>>> Yes, I set it to negative 60.
>>>>
>>>> It's not a problem if the session is killed. That's actually what I try
>>>> to do, because I can't allow to a user to try to end an infinite request.
>>>> Therefore I'll try your solution :)
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> Loïc
>>>>
>>>> Loïc CHANEL
>>>> Engineering student at TELECOM Nancy
>>>> Trainee at Worldline - Villeurbanne
>>>>
>>>> 2015-07-29 16:14 GMT+02:00 Xuefu Zhang <xzhang@cloudera.com>:
>>>>
>>>>> Okay. To confirm, you set it to negative 60s?
>>>>>
>>>>> The next thing you can try is to set
>>>>> hive.server2.idle.session.timeou=60000 (60sec) and
>>>>> hive.server2.idle.session.check.operation=false. I'm pretty sure this
>>>>> works, but the user's session will be killed though.
>>>>>
>>>>> --Xuefu
>>>>>
>>>>> On Wed, Jul 29, 2015 at 7:02 AM, Loïc Chanel <
>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>
>>>>>> I confirm : I just tried hive.server2.idle.operation.timeout setting
>>>>>> it to -60 (seconds), but my veeeeeery slow job have not been killed.
The
>>>>>> issue here is "what if another user come and try to submit a MapReduce
job
>>>>>> but the cluster is stuck in an infinite loop ?".
>>>>>>
>>>>>> Do you or anyone else have another idea ?
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>> Loïc
>>>>>>
>>>>>> Loïc CHANEL
>>>>>> Engineering student at TELECOM Nancy
>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>
>>>>>> 2015-07-29 15:34 GMT+02:00 Loïc Chanel <loic.chanel@telecomnancy.net>
>>>>>> :
>>>>>>
>>>>>>> No, because I thought the idea of infinite operation was not
very
>>>>>>> compatible with the "idle" word (as the operation will not stop
running),
>>>>>>> but I'll try :-)
>>>>>>> Thanks for the idea,
>>>>>>>
>>>>>>>
>>>>>>> Loïc
>>>>>>>
>>>>>>> Loïc CHANEL
>>>>>>> Engineering student at TELECOM Nancy
>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>
>>>>>>> 2015-07-29 15:27 GMT+02:00 Xuefu Zhang <xzhang@cloudera.com>:
>>>>>>>
>>>>>>>> Have you tried hive.server2.idle.operation.timeout?
>>>>>>>>
>>>>>>>> --Xuefu
>>>>>>>>
>>>>>>>> On Wed, Jul 29, 2015 at 5:52 AM, Loïc Chanel <
>>>>>>>> loic.chanel@telecomnancy.net> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> As I'm trying to build a secured and multi-tenant Hadoop
cluster
>>>>>>>>> with Hive, I am desperately trying to set a timeout to
Hive requests.
>>>>>>>>> My idea is that some users can make mistakes such as
a join with
>>>>>>>>> wrong keys, and therefore start an infinite loop believing
that they are
>>>>>>>>> just launching a very heavy job. Therefore, I'd like
to set a limit to the
>>>>>>>>> time a request should take, in order to kill the job
automatically if it
>>>>>>>>> exceeds it.
>>>>>>>>>
>>>>>>>>> As such a notion cannot be set directly in YARN, I saw
that
>>>>>>>>> MapReduce2 provides with its own native timeout property,
and I would like
>>>>>>>>> to know if Hive provides with the same property someway.
>>>>>>>>>
>>>>>>>>> Did anyone heard about such a thing ?
>>>>>>>>>
>>>>>>>>> Thanks in advance for your help,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Loïc
>>>>>>>>>
>>>>>>>>> Loïc CHANEL
>>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message