hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
Date Tue, 24 Apr 2012 03:09:13 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260172#comment-13260172

Siddharth Seth commented on MAPREDUCE-3921:

Couple of questions and suggestions.
- Does 'node unhealthy' need to be treated differently from 'TooManyFetchFailures' ? Killed
versus Failed. This ends up with NodeFailures not counting towards the limit on task attempts.
- JOB_UPDATED_NODES needs to be handled in the JOB_INIT state. Very small chance of hitting
- Minor: JobImpl.actOnUsableNode can get the task type from the id itself. It doesn't need
to fetch the actual task.
- Minor: "if this attempt is not successful" this comment in JobImpl can be removed. It's
removing an entry from a successfulAttempt index.
- In KilledAfterSuccessTransition - createJobCounterUpdateEventTAFailed should be createJobCounterUpdateEventTAKilled
- TaskImpl.handleAttemptCompletion - finishedAttempts - this will end up double counting the
same task attempt. It's used in some other transition.
- Does the JobHistoryParser need some more changes - to unset fields which may have been set
previously by Map/ReduceAttemptSuccessfulEvents and TaskFinishedEvent
- For running tasks - shouldn't running Reduce attempts also be killed ?
- RMContainerAllocator.handleUpdatedNodes - instead of fetching the nodeId via appContext,
job etc - the nodeId can be stored with the AssignedRequest. 1) getTask, getAttempt require
readLocks - can avoid these calls every second. 2) There's an unlikely race where the nodeId
may not be assigned in the TaskAttempt (if the dispatcher thread is backlogged).
- TaskAttemptId.getNodeId() can be avoided. getContainerManagerAddress can be used instead.

Not related to this patch.
Does JOB_TASK_COMPLETED need to be handled (ignored) in additional states?
> MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
> ------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-3921
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mr-am, mrv2
>    Affects Versions: 0.23.0
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Bikas Saha
>             Fix For: 0.23.2
>         Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, MAPREDUCE-3921-4.patch,
MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch,

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message