hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1325) DFSClient(DFSInputStream) release the persistent connection with datanode when no data have been read for a long time
Date Thu, 05 Aug 2010 21:43:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895821#action_12895821
] 

Suresh Srinivas commented on HDFS-1325:
---------------------------------------

Clients that run for short duration of time, when exits, releases connection resources. Long
running clients, typically another server talking to HDFS (such as HBASE), should close the
file when it is done. The change you are introducing results in fostering poor code and masks
the root cause of the problem.

Inspite of this, if you feel that this is the way you want to solve the problem, please consider
adding this check into the application layer code, instead of DFSClient.

> DFSClient(DFSInputStream) release the persistent connection with datanode when no data
have been read for a long time
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1325
>                 URL: https://issues.apache.org/jira/browse/HDFS-1325
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs client
>            Reporter: jinglong.liujl
>             Fix For: 0.20.3
>
>         Attachments: dfsclient.patch, toomanyconnction.patch
>
>
> When you use Hbase over hadoop. We found during scanning over a large table ( which has
many regions and each region has many store files), there're too many connections has been
kept between regionserver (act as DFSClient) and datanode.  Even if the store file has been
complete to scanning, the connections can not be closed.
> In our cluster, too many extra connections cause too many system resource has been wasted,
which cause system cpu on region server reach to a high level, then bring this region server
down.
> After investigating, we found the number of active connection is very small, and the
most connection is idle. We add a timeout checker thread into DFSClient, to close this connection.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message