hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6826) Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
Date Wed, 11 Apr 2018 16:21:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434128#comment-16434128
] 

Hadoop QA commented on MAPREDUCE-6826:
--------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 17s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  0s{color} | {color:red}
The patch doesn't appear to include any new or modified tests. Please justify why no new tests
are needed for this patch. Also please list what manual steps were performed to verify this
patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 33s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m 47s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 40s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 18m 36s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 29m 53s{color}
| {color:green} branch has no errors when building and testing our client artifacts. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  0s{color} | {color:blue}
Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m  0s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  4m 57s{color} |
{color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 25s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m 42s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 25m 42s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 47s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 19m  4s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  9m 19s{color}
| {color:green} patch has no errors when building and testing our client artifacts. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  0s{color} | {color:blue}
Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m  0s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  5m 17s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}127m 43s{color} | {color:red}
root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 36s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}299m 18s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
|   | hadoop.hdfs.TestEncryptionZonesWithKMS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | MAPREDUCE-6826 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12914665/MAPREDUCE-6826-001.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient
 findbugs  checkstyle  |
| uname | Linux 9ec1abd78009 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7eb783e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| unit | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7385/artifact/out/patch-unit-root.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7385/testReport/
|
| Max. process+thread count | 4796 (vs. ulimit of 10000) |
| modules | C: . U: . |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7385/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
> ---------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6826
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6826
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.7.2
>            Reporter: Varun Saxena
>            Assignee: Bilwa S T
>            Priority: Major
>         Attachments: MAPREDUCE-6826-001.patch
>
>
> This happens if a container is preempted by scheduler after job starts committing.
> And this exception in turn leads to application being marked as FAILED in YARN.
> I think we can probably ignore JOB_TASK_COMPLETED event while JobImpl state is COMMITTING
or SUCCEEDED as job is in the process of finishing.
> Also is there any point in attempting to scheduler another task attempt if job is already
in COMMITTING or SUCCEEDED state.
> {noformat}
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
task_1482404625971_23910_m_000004 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:38,642 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
Num completed Tasks: 5
> 2016-12-23 09:10:38,643 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
job_1482404625971_23910Job Transitioned from RUNNING to COMMITTING
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e55_1482404625971_23910_01_000010
taskAttempt attempt_1482404625971_23910_m_000004_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
KILLING attempt_1482404625971_23910_m_000004_1
> 2016-12-23 09:10:38,644 INFO [ContainerLauncher #5] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : linux-19:26009
> 2016-12-23 09:10:38,644 INFO [CommitterEvent Processor #4] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler:
Processing the event EventType: JOB_COMMIT
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
JVM with ID : jvm_1482404625971_23910_m_60473139527690 asked for a task
> 2016-12-23 09:10:38,724 INFO [IPC Server handler 0 on 27113] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
JVM with ID: jvm_1482404625971_23910_m_60473139527690 is invalid and will be killed.
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
Calling handler for JobFinishedEvent 
> 2016-12-23 09:10:38,797 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
job_1482404625971_23910Job Transitioned from COMMITTING to SUCCEEDED
> 2016-12-23 09:10:38,798 INFO [Thread-93] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Job finished cleanly, recording last MRAppMaster retry
> 2016-12-23 09:10:38,798 INFO [Thread-93] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Notify RMCommunicator isAMLastRetry: true
> 2016-12-23 09:10:38,798 INFO [Thread-93] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator:
RMCommunicator notified that shouldUnregistered is: true
> 2016-12-23 09:10:38,799 INFO [Thread-93] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Notify JHEH isAMLastRetry: true
> 2016-12-23 09:10:38,799 INFO [Thread-93] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:38,799 INFO [Thread-93] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Calling stop for all the services
> 2016-12-23 09:10:38,800 INFO [Thread-93] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
Stopping JobHistoryEventHandler. Size of the outstanding queue size is 1
> 2016-12-23 09:10:38,989 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0
CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:0 RackLocal:0
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Received completed container container_e55_1482404625971_23910_01_000010
> 2016-12-23 09:10:38,993 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0
CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:0 RackLocal:0
> 2016-12-23 09:10:38,993 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
Diagnostics report from attempt_1482404625971_23910_m_000004_1: Container preempted by scheduler
> 2016-12-23 09:10:38,994 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1482404625971_23910_m_000004_1 TaskAttempt Transitioned from SUCCEEDED to KILLED
> 2016-12-23 09:10:38,995 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
task_1482404625971_23910_m_000004 Task Transitioned from SUCCEEDED to SCHEDULED
> 2016-12-23 09:10:38,996 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1482404625971_23910_m_000004_2 TaskAttempt Transitioned from NEW to UNASSIGNED
> 2016-12-23 09:10:39,044 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0
CompletedMaps:5 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:0 RackLocal:0
> 2016-12-23 09:10:39,050 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
getResources() for application_1482404625971_23910: ask=1 release= 0 newContainers=0 finishedContainers=0
resourcelimit=<memory:0, vCores:0> knownNMs=11
> 2016-12-23 09:10:40,053 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Got allocated containers 1
> 2016-12-23 09:10:40,055 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Assigned container container_e55_1482404625971_23910_01_000011 to attempt_1482404625971_23910_m_000004_2
> 2016-12-23 09:10:40,055 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0
CompletedMaps:5 CompletedReds:0 ContAlloc:9 ContRel:0 HostLocal:0 RackLocal:0
> 2016-12-23 09:10:40,056 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1482404625971_23910_m_000004_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
> 2016-12-23 09:10:40,056 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e55_1482404625971_23910_01_000011
taskAttempt attempt_1482404625971_23910_m_000004_2
> 2016-12-23 09:10:40,056 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
Launching attempt_1482404625971_23910_m_000004_2
> 2016-12-23 09:10:40,056 INFO [ContainerLauncher #6] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : linux-17:26009
> 2016-12-23 09:10:40,068 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
Shuffle port returned by ContainerManager for attempt_1482404625971_23910_m_000004_2 : 26008
> 2016-12-23 09:10:40,074 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
TaskAttempt: [attempt_1482404625971_23910_m_000004_2] using containerId: [container_e55_1482404625971_23910_01_000011
on NM: [linux-17:26009]
> 2016-12-23 09:10:40,074 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1482404625971_23910_m_000004_2 TaskAttempt Transitioned from ASSIGNED to RUNNING
> 2016-12-23 09:10:40,075 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
task_1482404625971_23910_m_000004 Task Transitioned from SCHEDULED to RUNNING
> 2016-12-23 09:10:40,108 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
getResources() for application_1482404625971_23910: ask=1 release= 0 newContainers=0 finishedContainers=0
resourcelimit=<memory:0, vCores:0> knownNMs=11
> 2016-12-23 09:10:42,949 INFO [Socket Reader #1 for port 27113] SecurityLogger.org.apache.hadoop.ipc.Server:
Auth successful for job_1482404625971_23910 (auth:SIMPLE)
> 2016-12-23 09:10:42,995 INFO [IPC Server handler 6 on 27113] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
JVM with ID : jvm_1482404625971_23910_m_60473139527691 asked for a task
> 2016-12-23 09:10:42,995 INFO [IPC Server handler 6 on 27113] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
JVM with ID: jvm_1482404625971_23910_m_60473139527691 given task: attempt_1482404625971_23910_m_000004_2
> 2016-12-23 09:10:46,163 INFO [IPC Server handler 29 on 27113] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Commit-pending state update from attempt_1482404625971_23910_m_000004_2
> 2016-12-23 09:10:46,165 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1482404625971_23910_m_000004_2 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING
> 2016-12-23 09:10:46,165 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
attempt_1482404625971_23910_m_000004_2 given a go for committing the task output.
> 2016-12-23 09:10:46,166 INFO [IPC Server handler 9 on 27113] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Commit go/no-go request from attempt_1482404625971_23910_m_000004_2
> 2016-12-23 09:10:46,167 INFO [IPC Server handler 9 on 27113] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
Result of canCommit for attempt_1482404625971_23910_m_000004_2:true
> 2016-12-23 09:10:46,265 INFO [IPC Server handler 20 on 27113] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Done acknowledgement from attempt_1482404625971_23910_m_000004_2
> 2016-12-23 09:10:46,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1482404625971_23910_m_000004_2 TaskAttempt Transitioned from COMMIT_PENDING to SUCCEEDED
> 2016-12-23 09:10:46,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
Task succeeded with attempt attempt_1482404625971_23910_m_000004_2
> 2016-12-23 09:10:46,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
task_1482404625971_23910_m_000004 Task Transitioned from RUNNING to SUCCEEDED
> 2016-12-23 09:10:46,267 ERROR [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED
at SUCCEEDED
> 	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> 	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> 	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:997)
> 	at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:139)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1399)
> 	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1395)
> 	at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:192)
> 	at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:118)
> 	at java.lang.Thread.run(Thread.java:745)
> 2016-12-23 09:10:46,270 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
job_1482404625971_23910Job Transitioned from SUCCEEDED to ERROR
> 2016-12-23 09:10:46,271 INFO [Thread-97] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Notify RMCommunicator isAMLastRetry: true
> 2016-12-23 09:10:46,271 INFO [Thread-97] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator:
RMCommunicator notified that shouldUnregistered is: true
> 2016-12-23 09:10:46,271 INFO [Thread-97] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Notify JHEH isAMLastRetry: true
> 2016-12-23 09:10:46,271 INFO [Thread-97] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
JobHistoryEventHandler notified that forceJobCompletion is true
> 2016-12-23 09:10:46,271 INFO [Thread-97] org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Calling stop for all the services
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message