hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@yahoo-inc.com>
Subject Re: "Too many open files" error, which gets resolved after some time
Date Mon, 22 Jun 2009 08:05:17 GMT

Is this before 0.20.0? Assuming you have closed these streams, it is 
mostly https://issues.apache.org/jira/browse/HADOOP-4346

It is the JDK internal implementation that depends on GC to free up its 
cache of selectors. HADOOP-4346 avoids this by using hadoop's own cache.

Raghu.

Stas Oskin wrote:
> Hi.
> 
> After tracing some more with the lsof utility, and I managed to stop the
> growth on the DataNode process, but still have issues with my DFS client.
> 
> It seems that my DFS client opens hundreds of pipes and eventpolls. Here is
> a small part of the lsof output:
> 
> java    10508 root  387w  FIFO                0,6           6142565 pipe
> java    10508 root  388r  FIFO                0,6           6142565 pipe
> java    10508 root  389u  0000               0,10        0  6142566
> eventpoll
> java    10508 root  390u  FIFO                0,6           6135311 pipe
> java    10508 root  391r  FIFO                0,6           6135311 pipe
> java    10508 root  392u  0000               0,10        0  6135312
> eventpoll
> java    10508 root  393r  FIFO                0,6           6148234 pipe
> java    10508 root  394w  FIFO                0,6           6142570 pipe
> java    10508 root  395r  FIFO                0,6           6135857 pipe
> java    10508 root  396r  FIFO                0,6           6142570 pipe
> java    10508 root  397r  0000               0,10        0  6142571
> eventpoll
> java    10508 root  398u  FIFO                0,6           6135319 pipe
> java    10508 root  399w  FIFO                0,6           6135319 pipe
> 
> I'm using FSDataInputStream and FSDataOutputStream, so this might be related
> to pipes?
> 
> So, my questions are:
> 
> 1) What happens these pipes/epolls to appear?
> 
> 2) More important, how I can prevent their accumation and growth?
> 
> Thanks in advance!
> 
> 2009/6/21 Stas Oskin <stas.oskin@gmail.com>
> 
>> Hi.
>>
>> I have HDFS client and HDFS datanode running on same machine.
>>
>> When I'm trying to access a dozen of files at once from the client, several
>> times in a row, I'm starting to receive the following errors on client, and
>> HDFS browse function.
>>
>> HDFS Client: "Could not get block locations. Aborting..."
>> HDFS browse: "Too many open files"
>>
>> I can increase the maximum number of files that can opened, as I have it
>> set to the default 1024, but would like to first solve the problem, as
>> larger value just means it would run out of files again later on.
>>
>> So my questions are:
>>
>> 1) Does the HDFS datanode keeps any files opened, even after the HDFS
>> client have already closed them?
>>
>> 2) Is it possible to find out, who keeps the opened files - datanode or
>> client (so I could pin-point the source of the problem).
>>
>> Thanks in advance!
>>
> 


Mime
View raw message