hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruba Borthakur <dhr...@gmail.com>
Subject Re: libhdfs with FileSystem cache issue can causes to memory leak ?
Date Tue, 13 Oct 2009 19:45:59 GMT
There is a python interface to access HDFS files if that helps your case :
http://wiki.apache.org/hadoop/HDFS-APIs

thanks,
dhruba

On Tue, Oct 13, 2009 at 12:00 PM, Huy Phan <dachuy@gmail.com> wrote:

> Hi Brian,
> Thank you for posting your solution here, I will try this on my testing
> server and do some load tests.
> Also thank you for pointing out some leaks inside libhdfs. Actually I'm
> writing a Python extension for HDFS and noticed some Memory Leaks, but I was
> not sure if it's the bug of my extension or somewhere else.
>
> Regards,
> Huy Phan
>
> Brian Bockelman wrote:
>
>> Hey Huy,
>>
>> Heres what we do:
>>
>> 1) include hdfsJniHelper.h
>> 2) Do the following when you're done with the filesystem:
>>
>>    if (NULL != fs) {
>>      //Get the JNIEnv* corresponding to current thread
>>      JNIEnv* env = getJNIEnv();
>>
>>      if (env == NULL) {
>>        ret = -EIO;
>>      } else {
>>
>>        //Parameters
>>        jobject jFS = (jobject)fs;
>>
>>        //Release unnecessary references
>>        (*env)->DeleteGlobalRef(env, jFS);
>>      }
>>    }
>>
>> I also recommend the below patch to remove a few other leaks.  This saves
>> about .5KB / file open in leaked memory.
>>
>> Index: src/c++/libhdfs/hdfs.c
>> ===================================================================
>> --- src/c++/libhdfs/hdfs.c      (revision 806186)
>> +++ src/c++/libhdfs/hdfs.c      (working copy)
>> @@ -248,6 +249,7 @@
>>       destroyLocalReference(env, jUserString);
>>       destroyLocalReference(env, jGroups);
>>       destroyLocalReference(env, jUgi);
>> +      destroyLocalReference(env, jAttrString);
>>     }
>>  #else
>>
>> Index: src/c++/libhdfs/hdfsJniHelper.c
>> ===================================================================
>> --- src/c++/libhdfs/hdfsJniHelper.c     (revision 806186)
>> +++ src/c++/libhdfs/hdfsJniHelper.c     (working copy)
>> @@ -239,6 +241,7 @@
>>       fprintf(stderr, "ERROR: jelem == NULL\n");
>>     }
>>     (*env)->SetObjectArrayElement(env, result, i, jelem);
>> +    (*env)->DeleteLocalRef(env, jelem);
>>   }
>>   return result;
>>  }
>>
>>
>> Of course, this is not an official solution, not supported, may explode,
>> etc.
>>
>> Brian
>>
>> On Oct 13, 2009, at 12:40 PM, Huy Phan wrote:
>>
>>  Hi Eli,
>>>  You're right that the problem is resolved in 0.20 with function
>>> newInstance(), unfortunately my system's running on Hadoop 0.18.3 and i'm
>>> still looking for a way to patch this version without affecting the current
>>> system.
>>>
>>> Regards,
>>> Huy Phan
>>>
>>> Eli Collins wrote:
>>>
>>>> Hey Huy,
>>>>
>>>> What version of hadoop are you using?  I think HADOOP-4655 may have
>>>> resolved the issue you're seeing but I think is only in 20 and later.
>>>>
>>>> Thanks,
>>>> Eli
>>>>
>>>> On Mon, Oct 12, 2009 at 8:52 PM, Huy Phan <dachuy@gmail.com> wrote:
>>>>
>>>>  Hi All,
>>>>>  I'm writing a multi-thread application using libhdfs in C, a known
>>>>> issue
>>>>> of HDFS is that the FileSystem API caches FileSystem handles and always
>>>>> returned the same FileSystem handle when called from different threads.
>>>>> It
>>>>> means even though I called hdfsConnect for many times, I should not
>>>>> call
>>>>> hdfsDisconnect in any single thread.
>>>>> This may lead to memory leak on system, do you know any workaround for
>>>>> this
>>>>> issue ?
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>
>


-- 
Connect to me at http://www.facebook.com/dhruba

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message