hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@yahoo-inc.com>
Subject Re: File Descriptors not cleaned up
Date Wed, 30 Jul 2008 23:36:12 GMT
Arv Mistry wrote:
>  
> Thanks for responding Raghu,
> 
> This code is run every hour, where I open a file ( a different file each
> time) and write it across the network to another location. So if
> everytime it adds an additional 3 fd's then after some time I'm going to
> run out of fd's

It should not add 3 fds every time. If you do see a practical case where 
you trace running out of fds to these three fds, please let us know.

Raghu.

> Cheers Arv
> 
> 
> -----Original Message-----
> From: Raghu Angadi [mailto:rangadi@yahoo-inc.com] 
> Sent: Wednesday, July 30, 2008 4:33 PM
> To: core-user@hadoop.apache.org
> Subject: Re: File Descriptors not cleaned up
> 
> Arv Mistry wrote:
>>  
>> I've been trying to track down an issue where after some time I get 
>> "Too many files open " i.e.
>> we're not cleaning up somewhere ...
>>
>> I'm using "lsof -p <pid>" to track the open files and I find it's 
>> adding
>> 3 file descriptors everytime I do a
>> fs.open(<file>) where fs is FileSystem and <file> is a Path object to

>> a gzipped file in hadoop. When I'm done I call
>> Close() on the FSDataInputStream that the open returned. But those 3 
>> file descriptors never get cleaned up.
>>
>> The 3 fd's; 2 are 'pipe' and 1 'eventpoll' everytime.
> 
> Thats ok. Hadoop I/O leaves this set. What makes you think this set of 3
> fds is causing "Too many file open"? I doubt it. Do you see many sets of
> these fds being left open?
> 
>> Is there some other cleanup method I should be calling, other than on 
>> the InputStream after the open()?
> 
> This is no API to clean up these last set of fds currently.
> 
> Raghu.
> 
>> I'm using hadoop-0.17.0 and have also tried hadoop-0.17.1
>>
>> Cheers Arv
> 


Mime
View raw message