hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pete Wyckoff (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-1422) tasktracker checkpointing capability
Date Wed, 23 May 2007 00:25:16 GMT
tasktracker checkpointing capability
------------------------------------

                 Key: HADOOP-1422
                 URL: https://issues.apache.org/jira/browse/HADOOP-1422
             Project: Hadoop
          Issue Type: New Feature
          Components: mapred
            Reporter: Pete Wyckoff
            Priority: Minor


This relates to allowing a resource manager (e.g., hadoop on demand) to grow and (rarely)
shrink jobs on the fly.

Growing is already supported. Shrinking could be done in 2 ways - (1) consider the machine
dead and allow speculative execution to take care of it or (2) moving the existing map outputs
from that machine somewhere else (another machine, dfs) - "task tracker checkpointing" 

In the case of IO only intensive jobs,  checkpointing the tasktracker doesn't do much for
you.  But, in the case of CPU or other scarce resource (e.g., a DB or Webpage cache...), the
checkpointing could be very useful.  The question is how often is this the case and how useful?







-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message