flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefan Richter (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (FLINK-6412) Stream has already been closed during job cancel
Date Fri, 05 May 2017 13:33:04 GMT

     [ https://issues.apache.org/jira/browse/FLINK-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Stefan Richter reassigned FLINK-6412:
-------------------------------------

    Assignee: Stefan Richter

> Stream has already been closed during job cancel
> ------------------------------------------------
>
>                 Key: FLINK-6412
>                 URL: https://issues.apache.org/jira/browse/FLINK-6412
>             Project: Flink
>          Issue Type: Bug
>          Components: State Backends, Checkpointing
>    Affects Versions: 1.2.0
>            Reporter: Andrey
>            Assignee: Stefan Richter
>
> Steps to reproduce:
> * create job with RocksDBStateBackend: env.setStateBackend(new RocksDBStateBackend(checkpointRoot));
> * run job
> * cancel job from the Web UI.
> In logs:
> {code}
> 2017-04-28 10:25:57,475 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Source: Custom Source (1/2) (05a3888ed2a232f234a10775826395a0) switched from DEPLOYING
to RUNNING. [Source: Custom Source (1/2)]
> 2017-04-28 10:25:57,476 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask   
       - Using user-defined state backend: RocksDB State Backend {isInitialized=false, configuredDbBasePaths=null,
initializedDbBasePaths=null, checkpointStreamBackend=File State Backend @ file:/flink/checkpoints/flinktests}.
[Map (2/2)]
> 2017-04-28 10:25:57,476 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask   
       - Using user-defined state backend: RocksDB State Backend {isInitialized=false, configuredDbBasePaths=null,
initializedDbBasePaths=null, checkpointStreamBackend=File State Backend @ file:/flink/checkpoints/flinktests}.
[Source: Custom Source (1/2)]
> ...
> 2017-04-28 10:26:29,793 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Triggering cancellation of task code Source: Custom Source (1/2) (05a3888ed2a232f234a10775826395a0).
[flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,794 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Attempting to cancel task Map (2/2) (bdb982b6ef47fe79b6ff5b96153c921e). [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,794 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Map (2/2) (bdb982b6ef47fe79b6ff5b96153c921e) switched from RUNNING to CANCELING.
[flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,796 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Triggering cancellation of task code Map (2/2) (bdb982b6ef47fe79b6ff5b96153c921e).
[flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:29,796 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Source: Custom Source (1/2) (05a3888ed2a232f234a10775826395a0) switched from CANCELING
to CANCELED. [Source: Custom Source (1/2)]
> 2017-04-28 10:26:29,797 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Freeing task resources for Source: Custom Source (1/2) (05a3888ed2a232f234a10775826395a0).
[Source: Custom Source (1/2)]
> 2017-04-28 10:26:29,798 INFO  org.apache.flink.core.fs.FileSystem                   
       - Ensuring all FileSystem streams are closed for Source: Custom Source (1/2) [Source:
Custom Source (1/2)]
> 2017-04-28 10:26:29,803 INFO  org.apache.flink.runtime.taskmanager.TaskManager      
       - Un-registering task and sending final execution state CANCELED to JobManager for
task Source: Custom Source (05a3888ed2a232f234a10775826395a0) [flink-akka.actor.default-dispatcher-2]
> 2017-04-28 10:26:39,608 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Attempting to fail task externally Map (2/2) (bdb982b6ef47fe79b6ff5b96153c921e).
[pool-14-thread-1]
> 2017-04-28 10:26:39,608 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask   
       - Could not properly clean up the async checkpoint runnable. [Canceler for Map (2/2)
(bdb982b6ef47fe79b6ff5b96153c921e).]
> java.lang.Exception: Could not properly cancel managed keyed state future.
> 	at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:91)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.cleanup(StreamTask.java:1010)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.close(StreamTask.java:995)
> 	at org.apache.flink.util.IOUtils.closeQuietly(IOUtils.java:262)
> 	at org.apache.flink.util.IOUtils.closeAllQuietly(IOUtils.java:251)
> 	at org.apache.flink.util.AbstractCloseableRegistry.close(AbstractCloseableRegistry.java:91)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.cancel(StreamTask.java:364)
> 	at org.apache.flink.runtime.taskmanager.Task$TaskCanceler.run(Task.java:1390)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Stream has already
been closed and discarded.
> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> 	at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:40)
> 	at org.apache.flink.runtime.state.StateUtil.discardStateFuture(StateUtil.java:79)
> 	at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:89)
> 	... 8 more
> Caused by: java.io.IOException: Stream has already been closed and discarded.
> 	at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.closeAndGetHandle(FsCheckpointStreamFactory.java:342)
> 	at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBSnapshotOperation.closeSnapshotStreamAndGetHandle(RocksDBKeyedStateBackend.java:580)
> 	at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$RocksDBSnapshotOperation.closeCheckpointStream(RocksDBKeyedStateBackend.java:410)
> 	at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$1.performOperation(RocksDBKeyedStateBackend.java:298)
> 	at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend$1.performOperation(RocksDBKeyedStateBackend.java:277)
> 	at org.apache.flink.runtime.io.async.AbstractAsyncIOCallable.call(AbstractAsyncIOCallable.java:72)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:37)
> 	at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:915)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	... 1 more
> 2017-04-28 10:26:39,608 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Task Map (2/2) is already in state CANCELING [pool-14-thread-1]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message