hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4474) JT should not iterate through all jobs in every heartbeat to find a cleanup or setup task
Date Tue, 21 Oct 2008 16:43:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641430#action_12641430
] 

Devaraj Das commented on HADOOP-4474:
-------------------------------------

Yes, a valid point - O(#jobs) is certainly worse than O(1). But note that the iteration is
done only when a TT asks for a task and not per a heartbeat.

> JT should not iterate through all jobs in every heartbeat to find a cleanup or setup
task
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4474
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4474
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Vivek Ratan
>
> On every heartbeat, the JT first looks to see if it can run a setup or cleanup task,
before calling a Scheduler to get a Map or Reduce task. The JT maintains a hashmap of JobInProgress
objects (which can be waiting, running, or completed). It iterates through this hashmap on
each heartbeat to find a setup or cleanup task. This linear search can be be very expensive,
especially with large clusters where the number of jobs is high. There are lots of obvious
ways to cut down on this linear search. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message