hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3140) JobTracker should not try to promote a (map) task if it does not write to DFS at all
Date Tue, 01 Apr 2008 15:26:24 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584183#action_12584183

Amar Kamat commented on HADOOP-3140:

How about this
1) {{Task.done()}} method checks if the task has data to be promoted and passes this info
to the TaskTracker via the {{TaskTracker.done()}} api.
2) If there is no data to promote, the TaskTracker sets the task status as {{SUCCEEDED}} or
{{FAILED}} depending on whether the task succeeds or fails.
3) JobInProgress adds only {{COMMIT_PENDING}} tasks to the commit-pending queue. The commit-pending
queue deals with {{KILLED/FAILED}} tasks only if the commit-pending thread fails to save the
task output or if the TaskTracker is lost.
4) Temporary data from {{FAILED/KILLED}} tasks will be deleted once the job completes (see
5) {{JobInProgress.updateTaskStatus()}} can now be called with {{SUCCEEDED}} state from TaskTracker
(via heartbeat) or from the commit-pending queue.
5) If a JobInProgress.updateTaskStatus() is called with {{SUCCEEDED}} state for a completed
TIP it will be marked as {{KILLED}}.

> JobTracker should not try to promote a (map) task if it does not write to DFS at all
> ------------------------------------------------------------------------------------
>                 Key: HADOOP-3140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3140
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Runping Qi
> In most cases, map tasks do not write to dfs.
> Thus, when they complete, they should not be put into commit_pending queue at all.
> This will improve the task promotion significantly.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message