hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arv Mistry" <...@kindsight.net>
Subject RE: File Descriptors not cleaned up
Date Thu, 31 Jul 2008 13:29:42 GMT
 
I've simplified the code into a simple test harness with just hadoop
(see attached file)

I found that I can only reproduce this problem when I am doing the
fs.open() in a different thread. Even though in that same thread I am
doing a close().

Cheers Arv

-----Original Message-----
From: Raghu Angadi [mailto:rangadi@yahoo-inc.com] 
Sent: Wednesday, July 30, 2008 7:36 PM
To: core-user@hadoop.apache.org
Subject: Re: File Descriptors not cleaned up

Arv Mistry wrote:
>  
> Thanks for responding Raghu,
> 
> This code is run every hour, where I open a file ( a different file 
> each
> time) and write it across the network to another location. So if 
> everytime it adds an additional 3 fd's then after some time I'm going 
> to run out of fd's

It should not add 3 fds every time. If you do see a practical case where
you trace running out of fds to these three fds, please let us know.

Raghu.

> Cheers Arv
> 
> 
> -----Original Message-----
> From: Raghu Angadi [mailto:rangadi@yahoo-inc.com]
> Sent: Wednesday, July 30, 2008 4:33 PM
> To: core-user@hadoop.apache.org
> Subject: Re: File Descriptors not cleaned up
> 
> Arv Mistry wrote:
>>  
>> I've been trying to track down an issue where after some time I get 
>> "Too many files open " i.e.
>> we're not cleaning up somewhere ...
>>
>> I'm using "lsof -p <pid>" to track the open files and I find it's 
>> adding
>> 3 file descriptors everytime I do a
>> fs.open(<file>) where fs is FileSystem and <file> is a Path object to

>> a gzipped file in hadoop. When I'm done I call
>> Close() on the FSDataInputStream that the open returned. But those 3 
>> file descriptors never get cleaned up.
>>
>> The 3 fd's; 2 are 'pipe' and 1 'eventpoll' everytime.
> 
> Thats ok. Hadoop I/O leaves this set. What makes you think this set of

> 3 fds is causing "Too many file open"? I doubt it. Do you see many 
> sets of these fds being left open?
> 
>> Is there some other cleanup method I should be calling, other than on

>> the InputStream after the open()?
> 
> This is no API to clean up these last set of fds currently.
> 
> Raghu.
> 
>> I'm using hadoop-0.17.0 and have also tried hadoop-0.17.1
>>
>> Cheers Arv
> 


Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message