hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2384) Can MR make error response Immediately?
Date Sun, 31 Jul 2011 15:00:10 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073358#comment-13073358

Harsh J commented on MAPREDUCE-2384:

FWIW, the only place this change could interfere with is that OutputFormat#checkOutputSpecs()
can't no longer find distributed cache files on the JT FS. Don't think that really matters
since you can directly access stuff on the HDFS/LocalFS to workaround (as is the case with
how the DC is loaded).

> Can MR make error response Immediately?
> ---------------------------------------
>                 Key: MAPREDUCE-2384
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2384
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: job submission
>    Affects Versions: 0.21.0
>            Reporter: Denny Ye
>            Assignee: Harsh J
>             Fix For: 0.23.0
>         Attachments: MAPREDUCE-2384.r1.diff, MAPREDUCE-2384.r2.diff
> When I read the source code of MapReduce in Hadoop 0.21.0, sometimes it made me confused
about error response. For example:
>         1. JobSubmitter checking output for each job. MapReduce makes rule to limit that
each job output must be not exist to avoid fault overwrite. In my opinion, MR should verify
output at the point of client submitting. Actually, it copies related files to specified target
and then, doing the verifying. 
>         2. JobTracker.   Job has been submitted to JobTracker. In first step, JT create
JIT object that is very "huge" . Next step, JT start to verify job queue authority and memory
>         In normal case, verifying client input then response immediately if any cases
in fault. Regular logic can be performed if all the inputs have passed.  
>         It seems like that those code does not make sense for understanding. Is only
my personal opinion? Wish someone help me to explain the details. Thanks!

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message