hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3140) JobTracker should not try to promote a (map) task if it does not write to DFS at all
Date Mon, 31 Mar 2008 21:42:24 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Owen O'Malley updated HADOOP-3140:
----------------------------------

    Description: 
In most cases, map tasks do not write to dfs.
Thus, when they complete, they should not be put into commit_pending queue at all.
This will improve the task promotion significantly.

 

  was:

In most cases, map tasks do not write to dfs.
Thus, when they complete, they should not be put into commit_pending queue at all.
This will improve the task promotion significantly.

 

        Summary: JobTracker should not try to promote a (map) task if it does not write to
DFS at all  (was: JobTracker should not try to promote a (map) task if it dis not write to
DFS at all)

I think that the tasks should include a boolean in the done message to the task tracker that
says if they have output to promote. (And it should delete everything in the case of failure,
locally.) This is just an optimization. The framework (TaskTracker.Child.main) would look
in the work output directory and set true if there is anything to promote. The TT would then
set the state to commit-pending or success according to the flag value.

> JobTracker should not try to promote a (map) task if it does not write to DFS at all
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3140
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Runping Qi
>
> In most cases, map tasks do not write to dfs.
> Thus, when they complete, they should not be put into commit_pending queue at all.
> This will improve the task promotion significantly.
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message