hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From praveenesh kumar <praveen...@gmail.com>
Subject Re: Killing hadoop jobs automatically
Date Mon, 30 Jan 2012 09:07:53 GMT
@ Harsh -

Yeah, mapred.task.timeout is the valid option. but for some reasons, its
not happening the way it should be.. I am not sure what could be the
cause.Thing is my jobs are running fine, its just that they are slow at
shuffling phase, sometimes.. not everytime.. so I was thinking "as an admin
- can we control the running of jobs, just as a  test, where we can just
kill the jobs who are taking more time for execution -- not only those jobs
that are hanging..but jobs that are taking more execution time than
expected". Problems in my case is, end-user doesn't want to go through the
pain of managing/controlling jobs over hadoop. They want all these job
handling should happen automatically, so that made me to think in such a
way (which I know is not the best way)

Anyways, going away from the topic -- Is there anyway through which I can
improve my shuffling (through any configuration parameters only, knowing
the fact that users doesn't know the idea of minimizing the key/value
pairs)

Thanks,
Praveenesh

On Mon, Jan 30, 2012 at 1:06 PM, Masoud <masoud@agape.hanyang.ac.kr> wrote:

> Hi,
>
> Every Map/Reduce app has a Reporter, You can set the configuration
> parameter {mapred.task.timeout} of  Reporter to your desired value.
>
> Good Luck.
>
>
> On 01/30/2012 04:14 PM, praveenesh kumar wrote:
>
>> Yeah, I am aware of that, but it needs you to explicity monitor the job
>> and
>> look for jobid and then hadoop job -kill command.
>> What I want to know - "Is there anyway to do all this automatically by
>> providing some timer or something -- that if my job is taking more than
>> some predefined time, it would get killed automatically
>>
>> Thanks,
>> Praveenesh
>>
>> On Mon, Jan 30, 2012 at 12:38 PM, Prashant Kommireddi
>> <prash1784@gmail.com>wrote:
>>
>>  You might want to take a look at the kill command : "hadoop job -kill
>>> <jobid>".
>>>
>>> Prashant
>>>
>>> On Sun, Jan 29, 2012 at 11:06 PM, praveenesh kumar<praveenesh@gmail.com
>>>
>>>> wrote:
>>>> Is there anyway through which we can kill hadoop jobs that are taking
>>>> enough time to execute ?
>>>>
>>>> What I want to achieve is - If some job is running more than
>>>> "_some_predefined_timeout_**limit", it should be killed automatically.
>>>>
>>>> Is it possible to achieve this, through shell scripts or any other way ?
>>>>
>>>> Thanks,
>>>> Praveenesh
>>>>
>>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message