hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Olga Natkovich (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-2815) support for DeleteOnExit
Date Thu, 14 Feb 2008 01:47:08 GMT
support for DeleteOnExit

                 Key: HADOOP-2815
                 URL: https://issues.apache.org/jira/browse/HADOOP-2815
             Project: Hadoop Core
          Issue Type: New Feature
            Reporter: Olga Natkovich

Pig creates temp files that it wants to be removed at the end of the processing. The code
that removes the temp file is in the shutdown hook so that they get removed both under normal
shutdown as well as when process gets killed.

The problem that we are seeing is that by the time the code is called the DFS might already
be closed and the delete fails leaving temp files behind. Since we have no control over the
shutdown order, we have no way to make sure that the files get removed.

One way to solve this issue is to be able to mark the files as temp files so that hadoop can
remove them during its shutdown.

The stack trace I am seeing is

at org.apache.hadoop.dfs.DFSClient.checkOpen(DFSClient.java:158)
        at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:417)
        at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:144)
        at org.apache.pig.backend.hadoop.datastorage.HPath.delete(HPath.java:96)
        at org.apache.pig.impl.io.FileLocalizer$1.run(FileLocalizer.java:275)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message