hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sriramadasu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1354) Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS accesses
Date Tue, 16 Feb 2010 09:49:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834139#action_12834139

Amareshwari Sriramadasu commented on MAPREDUCE-1354:

Some more optimizations that can be done(though these do not effect scheduling code path):
* JobInProgress.getJobConf().getUser() is called from JobTracker.submitJob() and QueueManager.hasAccess().
Those calls can also be changed to job.getUser().
* JobInProgress.finishedMaps() and finishedReduces() are synchronized; they are called from
jobqueue_details.jsp which iterates through all jobs. If any job is in initialization, this
page doesn't come up until the initialization finishes.
* JobTracker.setJobPriority() should also check whether the job is initialized or not.

> Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS accesses
> ---------------------------------------------------------------------------------
>                 Key: MAPREDUCE-1354
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1354
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>            Reporter: Devaraj Das
>            Assignee: Arun C Murthy
>            Priority: Critical
>         Attachments: MAPREDUCE-1354_yhadoop20.patch, MAPREDUCE-1354_yhadoop20.patch,
MAPREDUCE-1354_yhadoop20.patch, MAPREDUCE-1354_yhadoop20.patch, MAPREDUCE-1354_yhadoop20.patch,
> It'd be nice to have the JobTracker object not be locked while accessing the HDFS for
reading the jobconf file and while writing the jobinfo file in the submitJob method. We should
see if we can avoid taking the lock altogether.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message