flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijiang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-5830) OutOfMemoryError during notify final state in TaskExecutor may cause job stuck
Date Fri, 17 Feb 2017 11:14:41 GMT
Zhijiang Wang created FLINK-5830:
------------------------------------

             Summary: OutOfMemoryError during notify final state in TaskExecutor may cause
job stuck
                 Key: FLINK-5830
                 URL: https://issues.apache.org/jira/browse/FLINK-5830
             Project: Flink
          Issue Type: Bug
            Reporter: Zhijiang Wang


The scenario is like this:

{{JobMaster}} tries to cancel all the executions when process failed execution, and the task
executor already acknowledge the cancel rpc message.
When notify the final state in {{TaskExecutor}}, it causes OOM in {{AkkaRpcActor}} and this
error is caught to log the info. The final state will not be sent any more.
The {{JobMaster}} can not receive the final state and trigger the restart strategy.

One solution is to catch the {{OutOfMemoryError}} and throw it, then it will cause to shut
down the {{ActorSystem}} resulting in exiting the {{TaskExecutor}}. The {{JobMaster}} can
be notified of {{TaskExecutor}} failure and fail all the tasks to trigger restart successfully.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message