hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-285) limit concurrent connections(data serving thread) in one datanode
Date Mon, 03 Feb 2014 11:16:11 GMT

     [ https://issues.apache.org/jira/browse/HDFS-285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Harsh J resolved HDFS-285.
--------------------------

    Resolution: Not A Problem

This has likely gone stale (probably addressed at a higher level via Raghu's earliest comments).

In having seen some pretty large HBase region sets on several clusters, and never having faced
the described stack limit OOME (but having faced the transceiver limits) I think this is likely
no longer an issue.

Closing out as 'Not a Problem' (anymore).

> limit concurrent connections(data serving thread) in one datanode
> -----------------------------------------------------------------
>
>                 Key: HDFS-285
>                 URL: https://issues.apache.org/jira/browse/HDFS-285
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Luo Ning
>            Priority: Minor
>
> i'm here after HADOOP-2341 and HADOOP-2346, in my hbase env, many opening mapfiles cause
datanode OOME(stack memory), because 2000+ data serving threads in datanode process.
> although HADOOP-2346 has implements timeouts, it will be some situation many connection
created  before the read timeout(default 6min) reach. like hbase does, it open all files on
regionserver startup. 
> limit concurrent connections(data serving thread) will make datanode more stable. and
i think it could be done in SocketIOWithTimeout$SelectorPool#select:
> 1. in SelectorPool#select, record all waiting SelectorInfo instances in a List at the
beginning, and remove it after 'Selector#select' done.
> 2. before real 'select',  do a limitation check, if reached, close the first selectorInfo.




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message