Return-Path: X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 22963E436 for ; Mon, 7 Jan 2013 18:24:14 +0000 (UTC) Received: (qmail 1090 invoked by uid 500); 7 Jan 2013 18:24:13 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 1037 invoked by uid 500); 7 Jan 2013 18:24:13 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 1024 invoked by uid 99); 7 Jan 2013 18:24:13 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jan 2013 18:24:13 +0000 Date: Mon, 7 Jan 2013 18:24:13 +0000 (UTC) From: "Todd Lipcon (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-9182) the buffer used in hdfsRead seems leaks when the thread exits MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13546119#comment-13546119 ] Todd Lipcon commented on HADOOP-9182: ------------------------------------- We don't do anything thread-local that could cause this. My guess is you're forgetting to close your files. If you can show a simple standalone reproducing program, we can take a look. Otherwise I'm inclined to mark "cannot reproduce" > the buffer used in hdfsRead seems leaks when the thread exits > ------------------------------------------------------------- > > Key: HADOOP-9182 > URL: https://issues.apache.org/jira/browse/HADOOP-9182 > Project: Hadoop Common > Issue Type: Bug > Components: filecache > Environment: Linux RHEP x64 > Reporter: dingyichuan > > I use multi-threads in my c++ program to download 3000 files in HDFS use libhdfs. Every thread is created by "pthread_create" to download a file and exit. We monitor the memory status when the program is running. It seems every thread will create a buffer which size is specified by the buffersize parameter in "hdfsOpenFile" function. But when the thread finish the task and exit, it doesn't free the buffer. So our program will eventually abort by Java's "out of memory" exception. I just don't know how to free the buffer or I use these functions in wrong way. Thanks! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira