hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-66) dfs client writes all data for a chunk to /tmp
Date Tue, 14 Mar 2006 05:40:46 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-66?page=all ]

Owen O'Malley updated HADOOP-66:
--------------------------------

    Attachment: tmp-delete.patch

There is a method on File to delete the file when the jvm exits. Here is a patch that calls
the method on the temporary block files. Apparently there is a bug in the jvm under windows
so that the files are only deleted if they are closed. For linux or solaris users, this should
make sure that no block files end up being dropped by the application.

> dfs client writes all data for a chunk to /tmp
> ----------------------------------------------
>
>          Key: HADOOP-66
>          URL: http://issues.apache.org/jira/browse/HADOOP-66
>      Project: Hadoop
>         Type: Bug
>   Components: dfs
>     Versions: 0.1
>     Reporter: Sameer Paranjpye
>     Assignee: Doug Cutting
>      Fix For: 0.1
>  Attachments: no-tmp.patch, tmp-delete.patch
>
> The dfs client writes all the data for the current chunk to a file in /tmp, when the
chunk is complete it is shipped out to the Datanodes. This can cause /tmp to fill up fast
when a lot of files are being written. A potentially better scheme is to buffer the written
data in RAM (application code can set the buffer size) and flush it to the Datanodes when
the buffer fills up.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message