hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Bockelman <bbock...@cse.unl.edu>
Subject Re: "Too many open files" error, which gets resolved after some time
Date Tue, 23 Jun 2009 11:49:24 GMT
Hey Stas,

It sounds like it's technically possible, but it also sounds like a  
horrible hack: I'd avoid this at all expense.  This is how cruft is  
born.

The pipes/epolls are something that eventually get cleaned up - but  
they don't get cleaned up often enough for your cluster.  I would  
recommend just increasing the limit on the node itself and then wait  
for an upgrade to "solve" this.

Brian

On Jun 23, 2009, at 3:31 AM, Stas Oskin wrote:

> Hi.
>
> Any idea if calling System.gc() periodically will help reducing the  
> amount
> of pipes / epolls?
>
> Thanks for your opinion!
>
> 2009/6/22 Stas Oskin <stas.oskin@gmail.com>
>
>> Ok, seems this issue is already patched in the Hadoop distro I'm  
>> using
>> (Cloudera).
>>
>> Any idea if I still should call GC manually/periodically to clean  
>> out all
>> the stale pipes / epolls?
>>
>> 2009/6/22 Steve Loughran <stevel@apache.org>
>>
>>> Stas Oskin wrote:
>>>
>>> Hi.
>>>>
>>>> So what would be the recommended approach to pre-0.20.x series?
>>>>
>>>> To insure each file is used only by one thread, and then it safe  
>>>> to close
>>>> the handle in that thread?
>>>>
>>>> Regards.
>>>>
>>>
>>> good question -I'm not sure. For anythiong you get with  
>>> FileSystem.get(),
>>> its now dangerous to close, so try just setting the reference to  
>>> null and
>>> hoping that GC will do the finalize() when needed
>>>
>>


Mime
View raw message