hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-312) Connections should not be cached
Date Fri, 01 Sep 2006 20:07:23 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-312?page=comments#action_12432226 ] 
            
Doug Cutting commented on HADOOP-312:
-------------------------------------

If CLOSE_CONNECTION is a constant, then it should be declared 'static final'.  But I think
it can be removed altogether, along with shouldCloseConnection.  Why not have the ConnectionCuller
simply set running=false, so that Connection.run() exits naturally and removes itself from
the connection cache?

The culler doesn't appear to remove the connection from the cache anyway.  It should, so that
new requests are not handed to a connection that is exiting.  And to further safeguard against
that, we can check, before calling connections.remove(address) that connections.get(address)
== this, rather than keeping a noException boolean.

ConnectionCuller's constructor should set its thread name.

MAX_RETRIES should be named 'maxRetries' and should get its value from a config parameter
named ipc.client.connect.max.retries.

Do we need a separate nocache config parameter, and code to support it?  Shouldn't this be
the same as simply setting maxidletime to 0?  If so, let's get rid of isCachingDisabled. 
This would give only one mode of operation, greatly reducing the number of possible bugs.

The culler should sleep for a minimum time (say 1 second), so it doesn't enter a tight loop.

ipc.server.system.somaxconn should be named something like ipc.server.listen.queue.size


> Connections should not be cached
> --------------------------------
>
>                 Key: HADOOP-312
>                 URL: http://issues.apache.org/jira/browse/HADOOP-312
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: no_conn_caching.patch, no_conn_caching.patch, no_conn_caching.patch,
no_conn_caching.patch, no_conn_caching.patch, no_conn_caching.patch, no_connection_caching.patch,
no_connection_caching.patch
>
>
> Servers and clients (client include datanodes, tasktrackers, DFSClients & tasks)
should not cache connections or maybe cache them for very short periods of time. Clients should
set up & tear down connections to the servers everytime they need to contact the servers
(including the heartbeats). If connection is cached, then reuse the existing connection for
a few subsequent transactions until the connection expires. The heartbeat interval should
be more so that many more clients (order of  tens of thousands) can be accomodated within
1 heartbeat interval.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message