hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1354) Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS accesses
Date Thu, 11 Feb 2010 10:08:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832437#action_12832437
] 

Hemanth Yamijala commented on MAPREDUCE-1354:
---------------------------------------------

One thing that was noticed was that the getCounters call in JobInProgress is synchronized.
The wrapper call to getCounters in Jobtracker acquires a lock on the JT and then calls JobInProgress.getCounters.
The problem is that if the job is being initialized under initTasks, then the jobtracker lock
can get held up. We saw an instance of this on our clusters. To avoid this case, one solution
could be to check if the job being queried is inited. This pattern is used in getTaskCompletionEvents.

> Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS accesses
> ---------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1354
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1354
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>            Reporter: Devaraj Das
>            Assignee: Arun C Murthy
>            Priority: Critical
>         Attachments: MAPREDUCE-1354_yhadoop20.patch
>
>
> It'd be nice to have the JobTracker object not be locked while accessing the HDFS for
reading the jobconf file and while writing the jobinfo file in the submitJob method. We should
see if we can avoid taking the lock altogether.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message