hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ankur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3818) Not possible to access a FileSystem from within a ShutdownHook
Date Mon, 28 Jul 2008 06:43:31 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617369#action_12617369

Ankur commented on HADOOP-3818:

The use case that we hit upon was client flushing data from Apache servers directly onto hadoop
clusters and client dying because of unexpected server failures in which case the client receives
SIGTERM and gets killed. What we want to do here is simply close our output stream before
exiting so that data isn't lost. 

But I can see other use cases where people might want to do additional operations in the client
like renaming, moving etc (besides deleting and closing).

So I see a couple of approaches for this. 
1. Provide additional API's like closeOnExit(..),  moveOnExit(...) etc. which I think would
not be so elegant and somewhat limited in the control they provide.

2. Provide additional API in FileSystem, something like FileSystem.addShutdownEventReciever()
that allows clients to register there custom cleanup code. In the shutdown hook, make sure
that all the exit handlers registered by client are invoked before any code in the shutdown
hook. This is more flexible but relies on the fact there is only one shutdown hook thread
that runs once JVM starts to shutdown which currently is the case,  I think. 

> Not possible to access a FileSystem from within a ShutdownHook
> --------------------------------------------------------------
>                 Key: HADOOP-3818
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3818
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.17.1
>            Reporter: Rowan Nairn
>            Priority: Minor
> FileSystem uses addShutdownHook to close all FileSystems at exit.  This makes it impossible
to access a FileSystem from within your own ShutdownHook threads, say for deleting incomplete
output.  Using a pre-existing FileSystem object is unsafe since it may be closed by the time
the thread executes.  Using FileSystem.get(...) results in an exception:
> Exception in thread "Thread-10" java.lang.IllegalStateException: Shutdown in progress
> 	at java.lang.Shutdown.add(Shutdown.java:81)
> 	at java.lang.Runtime.addShutdownHook(Runtime.java:190)
> 	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1293)
> 	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
> 	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:108)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message