hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Denny Ye (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-3362) Job always stay at 'Pending' status and cannot finish several days
Date Mon, 07 Nov 2011 06:59:52 GMT
Job always stay at 'Pending' status and cannot finish several days

                 Key: MAPREDUCE-3362
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3362
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: jobhistoryserver, jobtracker
    Affects Versions: 0.20.2
            Reporter: Denny Ye
            Priority: Critical

Our jobs are always keeping at 'pending' status several days. We checked jobtracker log and
found that one task(attemp) failed due to failure to store job history to HDFS. 

The issue begins from the business that another job remove the folder that this job is being
written with history log. In this case, there has 'ConcurrentModificationException' at JobHistory#log(ArrayList<PrintWriter>
writers, RecordTypes recordType, Keys[] keys, String[] values, JobID id). One thread checked
if there has any output error and removed output with history folder at HDFS has been removed,
another thread got 'ConcurrentModificationException' at current 'writers' is blank.

Unfortunately, no one want to catch this exception and it go thought to TaskTracker(it jump
over the calculating part to add 'finishedMapTask'). Then, another task(attemp) runs from
'failedMap' successfully, but the total 'finishedMapTask' number is not the all finishedMapTask.
JobCleanupTask cannot startup and job always stay at 'pending' status.

The root cause:
First task(attemp) failed with exception and this task add to 'failedMap' with decrease the
'finishedMap' counter. Next task(attemp) runs successfully and increase one for 'finishedMap'.
Due to failure the total 'finishedMap' is less that actual finishedMap counter, so the cleanup
task cannot runs. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message