hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-2118) optimize getJobSetupAndCleanupTasks
Date Wed, 07 Sep 2011 08:15:11 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Arun C Murthy updated MAPREDUCE-2118:
-------------------------------------

    Status: Open  (was: Patch Available)

Sorry to come in late, the patch has gone stale. Can you please rebase? Thanks.

Given this is not an issue with MRv2 should we still commit this? I'm happy to, but not sure
it's useful. Thanks.

> optimize getJobSetupAndCleanupTasks 
> ------------------------------------
>
>                 Key: MAPREDUCE-2118
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2118
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Joydeep Sen Sarma
>            Assignee: Joydeep Sen Sarma
>         Attachments: mapreduce-2118.1.patch, mapreduce.2118.2.patch
>
>
> in every heartbeat, while holding the JobTracker global lock, all jobs are scanned for
job setup/cleanup, task setup/cleanup. on a large system with many trackers (and heartbeats)
and many jobs - this becomes the bottleneck for JT throughput.
> One possible route may be to rework the code to not require the JT lock while asking
the JIP whether it has a setup/cleanup task. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message