flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anton Solovev (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5345) IOManager failed to properly clean up temp file directory
Date Mon, 16 Jan 2017 14:13:26 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824013#comment-15824013
] 

Anton Solovev commented on FLINK-5345:
--------------------------------------

[~rmetzger] [~StephanEwen] How about using {{#deleteQuietly}} and if a folder can't be deleted
- log a warn. Because it's a temporary file so theoretically the directory sooner or later
will be deleted by one of the another thread.


> IOManager failed to properly clean up temp file directory
> ---------------------------------------------------------
>
>                 Key: FLINK-5345
>                 URL: https://issues.apache.org/jira/browse/FLINK-5345
>             Project: Flink
>          Issue Type: Bug
>    Affects Versions: 1.1.3
>            Reporter: Robert Metzger
>            Assignee: Anton Solovev
>              Labels: simplex, starter
>             Fix For: 1.2.0, 1.3.0
>
>
> While testing 1.1.3 RC3, I have the following message in my log:
> {code}
> 2016-12-15 14:46:05,450 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask   
       - Timer service is shutting down.
> 2016-12-15 14:46:05,452 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Source: control events generator (29/40) (73915a232ba09e642f9dff92f8c8773a) switched
from CANCELING to CANCELED.
> 2016-12-15 14:46:05,452 INFO  org.apache.flink.runtime.taskmanager.Task             
       - Freeing task resources for Source: control events generator (29/40) (73915a232ba09e642f9dff92f8c8773a).
> 2016-12-15 14:46:05,454 INFO  org.apache.flink.yarn.YarnTaskManager                 
       - Un-registering task and sending final execution state CANCELED to JobManager for
task Source: control events genera
> tor (73915a232ba09e642f9dff92f8c8773a)
> 2016-12-15 14:46:40,609 INFO  org.apache.flink.yarn.YarnTaskManagerRunner           
       - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
> 2016-12-15 14:46:40,611 INFO  org.apache.flink.runtime.blob.BlobCache               
       - Shutting down BlobCache
> 2016-12-15 14:46:40,724 WARN  akka.remote.ReliableDeliverySupervisor                
       - Association with remote system [akka.tcp://flink@10.240.0.34:33635] has failed, address
is now gated for [5000] ms.
>  Reason is: [Disassociated].
> 2016-12-15 14:46:40,808 ERROR org.apache.flink.runtime.io.disk.iomanager.IOManager  
       - IOManager failed to properly clean up temp file directory: /yarn/nm/usercache/robert/appcache/application_148129128
> 9979_0024/flink-io-f0ff3f66-b9e2-4560-881f-2ab43bc448b5
> java.lang.IllegalArgumentException: /yarn/nm/usercache/robert/appcache/application_1481291289979_0024/flink-io-f0ff3f66-b9e2-4560-881f-2ab43bc448b5/62e14e1891fe1e334c921dfd19a32a84/StreamMap_11_24/dummy_state
does not exist
>         at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1637)
>         at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
>         at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
>         at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
>         at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
>         at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
>         at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
>         at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
>         at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
>         at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
>         at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
>         at org.apache.flink.runtime.io.disk.iomanager.IOManager.shutdown(IOManager.java:109)
>         at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.shutdown(IOManagerAsync.java:185)
>         at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$1.run(IOManagerAsync.java:105)
> {code}
> This was the last message logged from that machine. I suspect two threads are trying
to clean up the directories during shutdown?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message