hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pete Wyckoff (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-1422) tasktracker checkpointing capability
Date Wed, 23 May 2007 00:25:16 GMT
tasktracker checkpointing capability

                 Key: HADOOP-1422
                 URL: https://issues.apache.org/jira/browse/HADOOP-1422
             Project: Hadoop
          Issue Type: New Feature
          Components: mapred
            Reporter: Pete Wyckoff
            Priority: Minor

This relates to allowing a resource manager (e.g., hadoop on demand) to grow and (rarely)
shrink jobs on the fly.

Growing is already supported. Shrinking could be done in 2 ways - (1) consider the machine
dead and allow speculative execution to take care of it or (2) moving the existing map outputs
from that machine somewhere else (another machine, dfs) - "task tracker checkpointing" 

In the case of IO only intensive jobs,  checkpointing the tasktracker doesn't do much for
you.  But, in the case of CPU or other scarce resource (e.g., a DB or Webpage cache...), the
checkpointing could be very useful.  The question is how often is this the case and how useful?

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message