hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2561) /tmp/hadoop-${user}/dfs/tmp/tmp/client-${long}.tmp is not cleanup correctly
Date Wed, 16 Jan 2008 01:37:35 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559325#action_12559325
] 

Konstantin Shvachko commented on HADOOP-2561:
---------------------------------------------

Good catch.
It looks like the problem here is more general than what your solution does.
The client writes to the backup file for as long as it reaches the block size. So if the client
fails anywhere between 2 individual writes the backupFile will not be deleted. Simple try-catch
will not solve it. And there is not much you can do about it.
This issue should probably be resolved as a part of HADOOP-1707, which intends to eliminate
the local backup file completely.
Benjamin, could you please take a look at 1707 to make sure the solution works for you.

> /tmp/hadoop-${user}/dfs/tmp/tmp/client-${long}.tmp is not cleanup correctly
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-2561
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2561
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs, fs
>    Affects Versions: 0.14.0
>            Reporter: Benjamin Francisoud
>         Attachments: patch-DFSClient-HADOOP-2561.diff
>
>
> Diretory "/tmp/hadoop-${user}/dfs/tmp/tmp" is being filled with those kinfd of files:
client-226966559287638337420857.tmp
> I tried to look at the code and found:
> h3. DFSClient.java
> src/java/org/apache/hadoop/dfs/DFSClient.java
> {code:java}
> private void closeBackupStream() throws IOException {...}
> /* Similar to closeBackupStream(). Theoritically deleting a file
>  * twice could result in deleting a file that we should not.
>  */
> private void deleteBackupFile() {...}
> private File newBackupFile() throws IOException {
> String name = "tmp" + File.separator +
>                      "client-" + Math.abs(r.nextLong());
> File result = dirAllocator.createTmpFileForWrite(name,
>                                                        2 * blockSize,
>                                                        conf);
> return result;
> }
> {code}
> h3. LocalDirAllocator
> src/java/org/apache/hadoop/fs/LocalDirAllocator.java#AllocatorPerContext.java
> {code:java}
> /** Creates a file on the local FS. Pass size as -1 if not known apriori. We
>  *  round-robin over the set of disks (via the configured dirs) and return
>  *  a file on the first path which has enough space. The file is guaranteed
>  *  to go away when the JVM exits.
>  */
> public File createTmpFileForWrite(String pathStr, long size,
>         Configuration conf) throws IOException {
> // find an appropriate directory
> Path path = getLocalPathForWrite(pathStr, size, conf);
> File dir = new File(path.getParent().toUri().getPath());
> String prefix = path.getName();
> // create a temp file on this directory
> File result = File.createTempFile(prefix, null, dir);
> result.deleteOnExit();
> return result;
> }
> {code}
> First it seems to me it's a bit of a mess here I don't know if it's DFSClient.java#deleteBackupFile()
or LocalDirAllocator#createTmpFileForWrite() {deleteOnExit(); ) who is call ... or both. Why
not keep it dry and delete it only once.
> But the most important is the "deleteOnExit();" since it mean if it is never restarted
it will never delete files :(

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message