hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] [Reopened] (HDFS-3696) Create files with WebHdfsFileSystem goes OOM when file size is big
Date Wed, 25 Jul 2012 23:54:35 GMT

     [ https://issues.apache.org/jira/browse/HDFS-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tsz Wo (Nicholas), SZE reopened HDFS-3696:

0.23 needs a new patch since the test cannot be compiled.
> Create files with WebHdfsFileSystem goes OOM when file size is big
> ------------------------------------------------------------------
>                 Key: HDFS-3696
>                 URL: https://issues.apache.org/jira/browse/HDFS-3696
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.0.0-alpha
>            Reporter: Kihwal Lee
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Critical
>             Fix For: 0.23.3
>         Attachments: h3696_20120724.patch
> When doing "fs -put" to a WebHdfsFileSystem (webhdfs://), the FsShell goes OOM if the
file size is large. When I tested, 20MB files were fine, but 200MB didn't work.  
> I also tried reading a large file by issuing "-cat" and piping to a slow sink in order
to force buffering. The read path didn't have this problem. The memory consumption stayed
the same regardless of progress.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message