hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3416) deadlock in a job between map and reduce cores allocation
Date Mon, 30 Mar 2015 04:38:52 GMT

    [ https://issues.apache.org/jira/browse/YARN-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14386182#comment-14386182
] 

Rohith commented on YARN-3416:
------------------------------

bq. And then, a map fails and retry, waiting for a core, while the 300 reduces are waiting
for failed map to finish
When there is any failed maps, if all the reducers are ocupied the resources then ideally
the reducer pre emption should be triggered. AM logs would give some information about problem.

> deadlock in a job between map and reduce cores allocation 
> ----------------------------------------------------------
>
>                 Key: YARN-3416
>                 URL: https://issues.apache.org/jira/browse/YARN-3416
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 2.6.0
>            Reporter: mai shurong
>
> I submit a  big job, which has 500 maps and 350 reduce, to a queue(fairscheduler) with
300 max cores. When the big mapreduce job is running 100% maps, the 300 reduces have occupied
300 max cores in the queue. And then, a map fails and retry, waiting for a core, while the
300 reduces are waiting for failed map to finish. So a deadlock occur. As a result, the job
is blocked, and the later job in the queue cannot run because no available cores in the queue.
> I think there is the similar issue for memory of a queue .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message