hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Damien Hardy (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (MAPREDUCE-5019) Fair scheduler should allow peremption on reducer only
Date Fri, 12 Apr 2013 10:34:17 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Damien Hardy resolved MAPREDUCE-5019.
-------------------------------------

    Resolution: Not A Problem

set mapred.reduce.slowstart.completed.maps to higher value (0.8) is he right way to avoid
reducer blocking.
I close this issue
                
> Fair scheduler should allow peremption on reducer only
> ------------------------------------------------------
>
>                 Key: MAPREDUCE-5019
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5019
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv1, scheduler
>    Affects Versions: 2.0.2-alpha
>         Environment: CDH4.1.2
>            Reporter: Damien Hardy
>            Priority: Minor
>              Labels: scheduler, scheduling
>
> Fair scheduler is very good.
> But having a big MR job running lots of mapper and reducer( 10M + 10R )
> Then a small MR on the same pool (1M + 1R)
> having slots for 10 mapper and 10 reducer
>  - The big job take all the map slots
>  - The small job wait for a map slot
>  - 1rst big job map task finish
>  - the small job take the map slot it needs
>  - meanwhile all the reducer of the big job take all the reducer slot to copy and sort
>  - the small job end is map and wait for the all maps to end and for 1 reducer to end
before accessing for a reducer slot.
>  - all the reducer stalled after sorting waiting for the mapper to end one  by one...
> If I have a big job and a lot of small, I don't want new small arriving  and killing
running map tasks of big job to get a slot.
> I think it could be useful that the small job can kill a reducer tasks (and only reducer)
to end before the big job finish all its map tasks and a reducer.
> rules can be : a job having all its map finished and waiting for reducer slot can kill
reducer tasks from a job that still have map slot running (assuming they are just waiting
for copy and sort)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message