hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9182) the buffer used in hdfsRead seems leaks when the thread exits
Date Mon, 07 Jan 2013 18:50:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13546138#comment-13546138

Colin Patrick McCabe commented on HADOOP-9182:

Can you fill in the "Affects Version/s"?  This could be HDFS-799, which was fixed in branch-2
and later but never backported.

Also, can you please explain how you are "monitoring the memory status" (besides noticing
the out of memory exceptions).  Thanks.
> the buffer used in hdfsRead seems leaks when the thread exits
> -------------------------------------------------------------
>                 Key: HADOOP-9182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9182
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: filecache
>         Environment: Linux RHEP x64 
>            Reporter: dingyichuan
> I use multi-threads in my c++ program to download 3000 files in HDFS use libhdfs. Every
thread is created by "pthread_create" to download a file and exit. We monitor the memory status
when the program is running. It seems every thread will create a buffer which size is specified
by the buffersize parameter in "hdfsOpenFile" function. But when the thread finish the task
and exit, it doesn't free the buffer. So our program will eventually abort by Java's "out
of memory" exception. I just don't know how to free the buffer or I use these functions in
wrong way. Thanks!

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message