hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 정현진 <hyeonjin...@gmail.com>
Subject job list in yarn
Date Sat, 25 Mar 2017 05:10:42 GMT
Alexis,

You can get just finished jobs in specific range, using Cluster
Applications API.

http://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API

Setting finishedTimeBegin and finishedTimeEnd and request it periodically,
you don't have to get 5000 jobs but just get finished jobs in that range
every time.

For example, immediately after launching 5000 jobs,
  request { finishedTimeBegin 00:00, finishedTimeEnd 00:10 }
  sleep 10
  request { finishedTimeBegin 00:10, finishedTimeEnd 00:20 }
  sleep 10
... periodically


- Hyeonjin Jung



2017. 3. 25. 오전 1:15에 "alexis" <alzrck@gmail.com>님이 작성:
>
> actually i dont want to kill jobs.
> job list now contains 5000 entries. i do have a script to asks the
cluster api and gets the json  of this list, i know i can filter on this
request, what i need is to delete from the list older jobs so in every
request i dont have to fetch 5000 jobs info
> On Thu, Mar 23, 2017 at 12:33 AM, 정현진 <hyeonjin507@gmail.com> wrote:
>>
>> Alexis,
>> If you need to kill all running jobs, you can use this bash script.
>> ```bash
>> # get list of job's process IDs
>> JOB_LIST=$(hadoop job -list 2> /dev/null | grep job_ | awk '{print $1}')
>> # kill all jobs
>> for JOB in $JOB_LIST
>> do
>>         hadoop job -kill $JOB
>>         echo job (${JOB}) is killed now.
>> done
>> ```
>>
>> - Hyeonjin Jung
>> 2017. 3. 18. 오후 9:43에 "Alexis Fidalgo" <alzrck@gmail.com>님이 작성:
>>>
>>> Hello, is there any way to configure for this list (job list) to auto
clean or any command to clear without restart yarn?
>>>
>>> thanks
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: user-help@hadoop.apache.org
>>>
>
>

Mime
View raw message