hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4825) JobImpl.finished doesn't expect ERROR as a final job state
Date Thu, 29 Nov 2012 15:18:58 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506527#comment-13506527

Jason Lowe commented on MAPREDUCE-4825:

bq. Would there be a problem in metrics being notified of success/failure and then again of

Potentially, I forgot the job could leave these terminal states.  Some potential ways to address

* Don't allow the state to leave "terminal" states like SUCCEEDED/FAILED/KILLED.
* Add metrics for "errored" jobs to distinguish between failed and error.  This still means
that the sum of metrics could exceed the total number of job since a job can both succeed
and error.
* Have finished ignore incrementing any metrics if the job is already in a terminal state
(SUCCEEDED/FAILED/KILLED) to avoid double-counting a job.
> JobImpl.finished doesn't expect ERROR as a final job state
> ----------------------------------------------------------
>                 Key: MAPREDUCE-4825
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4825
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mr-am
>    Affects Versions: 2.0.3-alpha, 0.23.5
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>             Fix For: 3.0.0, 2.0.3-alpha, 0.23.6
>         Attachments: MAPREDUCE-4825.patch
> TestMRApp.testJobError is causing AsyncDispatcher to exit with System.exit due to an
exception being thrown.  From the console output from testJobError:
> {noformat}
> 2012-11-27 18:46:15,240 ERROR [AsyncDispatcher event handler] impl.TaskImpl (TaskImpl.java:internalError(665))
- Invalid event T_SCHEDULE on Task task_0_0000_m_000000
> 2012-11-27 18:46:15,242 FATAL [AsyncDispatcher event handler] event.AsyncDispatcher (AsyncDispatcher.java:dispatch(132))
- Error in dispatcher thread
> java.lang.IllegalArgumentException: Illegal job state: ERROR
> 	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.finished(JobImpl.java:838)
> 	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InternalErrorTransition.transition(JobImpl.java:1622)
> 	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$InternalErrorTransition.transition(JobImpl.java:1)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:359)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:299)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory.access$3(StateMachineFactory.java:287)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
> 	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:723)
> 	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:1)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:974)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1)
> 	at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:128)
> 	at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
> 	at java.lang.Thread.run(Thread.java:662)
> 2012-11-27 18:46:15,242 INFO  [AsyncDispatcher event handler] event.AsyncDispatcher (AsyncDispatcher.java:dispatch(135))
- Exiting, bbye..
> {noformat}

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message