hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-312) Connections should not be cached
Date Thu, 29 Jun 2006 18:07:30 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-312?page=comments#action_12418508 ] 

Devaraj Das commented on HADOOP-312:

The way I have designed this is to have the connection caching configurable. Given any IPC
server (like the namenode or jobtracker) the caching can range from fully cached (which is
the current behavior) to number of transactions per created connection. The configuration
related to caching can be set in the client's hadoop config file like:
  <value>server1:port=num_transactions, server2:port=num_transactions</value>
  <description>Defines the connections that should not be cached.
The above means that clients close the connections to server1:port after it has done num_transactions
(a transaction is defined as one request-response). Any number of such servers can be specified
by a comma-separated list. Connections to servers not explicitly mentioned in the config is
cached (though the server will disconnect clients that have not communicated with it for a
specified period of time).
Doing this will limit the number of connections cached significantly. For example, one can
always specify in the config that connections to the namenode should not be cached at all
(num_transactions = 1) and so on.
The patch is ready but I can probably incorporate any inputs you may have before I submit

> Connections should not be cached
> --------------------------------
>          Key: HADOOP-312
>          URL: http://issues.apache.org/jira/browse/HADOOP-312
>      Project: Hadoop
>         Type: Improvement

>   Components: ipc
>     Versions: 0.4.0
>     Reporter: Devaraj Das

> Servers and clients (client include datanodes, tasktrackers, DFSClients & tasks)
should not cache connections or maybe cache them for very short periods of time. Clients should
set up & tear down connections to the servers everytime they need to contact the servers
(including the heartbeats). If connection is cached, then reuse the existing connection for
a few subsequent transactions until the connection expires. The heartbeat interval should
be more so that many more clients (order of  tens of thousands) can be accomodated within
1 heartbeat interval.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message