hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@yahoo-inc.com>
Subject Re: xceiverCount limit reason
Date Thu, 08 Jan 2009 18:36:59 GMT
Jean-Adrien wrote:
> Is it the responsibility of hadoop client too manage its connection pool
> with the server ? In which case the problem would be an HBase problem?
> Anyway I found my problem, it is not a matter of performances.
> 

Essentially, yes. Client has to close the file to relinquish 
connections, if clients are using the common read/write interface.

Currently if a client keeps many hdfs files open, it results in many 
threads held at the DataNodes. As you noticed, timeout at DNs helps.

Various solutions are possible at different levels: application(hbase), 
Client API, HDFS, etc. https://issues.apache.org/jira/browse/HADOOP-3856 
is proposal at HDFS level.

Raghu.

Mime
View raw message