hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2231) ShellCommand, in particular 'df -k', sometimes hang
Date Tue, 04 Dec 2007 12:13:43 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12548237
] 

Amar Kamat commented on HADOOP-2231:
------------------------------------

After some more thoughts we feel that 
1. The way free space is computed using {{DF.getAvailable()}} should now be done using {{File.getFreeSpace()}}
and {{File.getUsableSpace()}} API available in java6. So as a part of this patch I will change
the way {{df -k}} is used to compute the free space by making use of {{File.getUsableSpace()}}.
2. There are some classes which still use the {{getMount()}} API of {{DF.java}}. Java 6 does
not provide any API for this. So the only way out is to make use of shell commands like {{stat}}
or {{df}} for calculating the mount points. As discussed in {{HADOOP-2344}}, there are issues
with the way shell commands are executed in {{ShellUtil.java}} and {{ShellCommand.java}}.
A patch is made available for {{HADOOP-2344}}. 
----
Comments ?

> ShellCommand, in particular 'df -k', sometimes hang
> ---------------------------------------------------
>
>                 Key: HADOOP-2231
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2231
>             Project: Hadoop
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.15.1
>            Reporter: Christian Kunz
>             Fix For: 0.15.2
>
>         Attachments: HADOOP-2231.patch
>
>
> We noticed that some pipes applications writing to dfs using libhdfs have about 6% chance
of hanging when executing 'df -k' to find out whether there is enough space available on the
local filesystem before opening a file for write.
> Why not using File.getFreeSpace() or File.GetUsableSpace()?
> The call stack is:
> Exception in thread "main" java.io.IOException
>          at org.apache.hadoop.fs.ShellCommand.runCommand
> (ShellCommand.java:52)
>          at org.apache.hadoop.fs.ShellCommand.run(ShellCommand.java:42)
>          at org.apache.hadoop.fs.DF.getAvailable(DF.java:72)
>          at org.apache.hadoop.fs.LocalDirAllocator
> $AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:264)
>          at org.apache.hadoop.fs.LocalDirAllocator
> $AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:294)
>          at
> org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite
> (LocalDirAllocator.java:155)
>          at org.apache.hadoop.dfs.DFSClient
> $DFSOutputStream.newBackupFile(DFSClient.java:1470)
>          at org.apache.hadoop.dfs.DFSClient
> $DFSOutputStream.openBackupStream(DFSClient.java:1437)
>          at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.writeChunk
> (DFSClient.java:1579)
>          at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk
> (FSOutputSummer.java:140)
>          at org.apache.hadoop.fs.FSOutputSummer.write1
> (FSOutputSummer.java:100)
>          at org.apache.hadoop.fs.FSOutputSummer.write
> (FSOutputSummer.java:86)
>          at org.apache.hadoop.fs.FSDataOutputStream
> $PositionCache.write(FSDataOutputStream.java:39)
>          at java.io.DataOutputStream.write(DataOutputStream.java:90)
>          at java.io.FilterOutputStream.write(FilterOutputStream.java:80)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message