hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-312) Connections should not be cached
Date Tue, 29 Aug 2006 20:55:23 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-312?page=comments#action_12431367 ] 
Doug Cutting commented on HADOOP-312:

This disables connection caching by default (and hence, everywhere, since no one yet overrides
this new option).  If we really don't think we need connection caching, then the client could
be radically simpler: no per-connection threads, etc.

I thought the original plan was not so much to disable all connection caching, but rather
limit it to hosts that have been contacted recently.  But I see no provision in the current
patch for timing out idle connections.

The incrementRef & decrementRef methods are each only called once.  In both cases the
check for isCachingDisabled can be skipped.  So these can simply become inUse++ and inUse--.

Let's not add a separate config option for the connect timeout, but rather just use the normal
io timeout.

Finallly, the socket close and open should not be within a synchronized(connections) block,
as, that way, a slow server will block access to all servers.

> Connections should not be cached
> --------------------------------
>                 Key: HADOOP-312
>                 URL: http://issues.apache.org/jira/browse/HADOOP-312
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: no_conn_caching.patch, no_conn_caching.patch, no_conn_caching.patch,
no_conn_caching.patch, no_conn_caching.patch, no_connection_caching.patch, no_connection_caching.patch
> Servers and clients (client include datanodes, tasktrackers, DFSClients & tasks)
should not cache connections or maybe cache them for very short periods of time. Clients should
set up & tear down connections to the servers everytime they need to contact the servers
(including the heartbeats). If connection is cached, then reuse the existing connection for
a few subsequent transactions until the connection expires. The heartbeat interval should
be more so that many more clients (order of  tens of thousands) can be accomodated within
1 heartbeat interval.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message